[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:19 dist-test-slave-fccw krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:19 dist-test-slave-fccw krb5kdc[395](info): set up 2 sockets
Sep 26 02:59:19 dist-test-slave-fccw krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 02:59:20.071825 388 mini_kdc.cc:121] Time spent starting KDC: real 2.087s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 02:59:20.367982 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.296s user 0.001s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 02:59:20.534056 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.166s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 02:59:20.757176 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.223s user 0.001s sys 0.003s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/test-user.keytab.
W20260926 02:59:21.051663 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.294s user 0.003s sys 0.000s
Sep 26 02:59:21 dist-test-slave-fccw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:21.124341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40807
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:21.440272 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:21.440537 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:21.440622 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:21.447489 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:21.447580 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:21.447625 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:21.447685 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:21.447732 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:21.456881 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:21.458091 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:21.459302 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:21.471711 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:21.471777 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:21.471777 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:21.472323 408 server_base.cc:1034] running on GCE node
I20260926 02:59:21.473129 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:21.474409 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:21.475623 408 hybrid_clock.cc:648] HybridClock initialized: now 1790391561475564 us; error 72 us; skew 500 ppm
I20260926 02:59:21.475965 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:21 dist-test-slave-fccw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391561, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:21.482488 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:21.483845 408 webserver.cc:460] Webserver started at http://127.0.97.62:37749/ using document root <none> and password file <none>
I20260926 02:59:21.484522 408 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:21.484660 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:21.484967 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:21.487810 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "3caa761ae98049339d5008f1236fdf2c"
format_stamp: "Formatted at 2026-09-26 02:59:21 on dist-test-slave-fccw"
server_key: "5b2bc4acad9033cbc9828f78ba498e3b"
server_key_iv: "a353bbace12b8a0c9eff1ddb64294c0e"
server_key_version: "encryptionkey@0"
I20260926 02:59:21.488550 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "3caa761ae98049339d5008f1236fdf2c"
format_stamp: "Formatted at 2026-09-26 02:59:21 on dist-test-slave-fccw"
server_key: "5b2bc4acad9033cbc9828f78ba498e3b"
server_key_iv: "a353bbace12b8a0c9eff1ddb64294c0e"
server_key_version: "encryptionkey@0"
I20260926 02:59:21.493510 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 02:59:21.496805 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:21.498343 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:59:21.498481 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "3caa761ae98049339d5008f1236fdf2c"
format_stamp: "Formatted at 2026-09-26 02:59:21 on dist-test-slave-fccw"
server_key: "5b2bc4acad9033cbc9828f78ba498e3b"
server_key_iv: "a353bbace12b8a0c9eff1ddb64294c0e"
server_key_version: "encryptionkey@0"
I20260926 02:59:21.498632 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:21.523175 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:21.527452 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:21.527683 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:21.546402 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40807
I20260926 02:59:21.546430 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40807 every 8 connection(s)
I20260926 02:59:21.547758 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:21.549566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 408
I20260926 02:59:21.549883 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:21.552299 388 external_mini_cluster.cc:1428] Setting key 7101ee8687ba19e1e3a8a5529063a411
I20260926 02:59:21.552516 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:21.563896 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c: Bootstrap starting.
I20260926 02:59:21.567109 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:21.568931 475 log.cc:824] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:21.571795 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c: No bootstrap required, opened a new log
I20260926 02:59:21.580487 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3caa761ae98049339d5008f1236fdf2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40807 } }
I20260926 02:59:21.580876 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:21.581029 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3caa761ae98049339d5008f1236fdf2c, State: Initialized, Role: FOLLOWER
I20260926 02:59:21.581681 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3caa761ae98049339d5008f1236fdf2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40807 } }
I20260926 02:59:21.581900 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:21.582036 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:21.582226 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 0 FOLLOWER]: Advancing to term 1
Sep 26 02:59:21 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:21.584904 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3caa761ae98049339d5008f1236fdf2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40807 } }
I20260926 02:59:21.585610 475 leader_election.cc:302] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3caa761ae98049339d5008f1236fdf2c; no voters:
I20260926 02:59:21.586330 475 leader_election.cc:288] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:21.586464 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:21.587692 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [term 1 LEADER]: Becoming Leader. State: Replica: 3caa761ae98049339d5008f1236fdf2c, State: Running, Role: LEADER
I20260926 02:59:21.588267 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3caa761ae98049339d5008f1236fdf2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40807 } }
I20260926 02:59:21.588605 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:21.593745 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3caa761ae98049339d5008f1236fdf2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3caa761ae98049339d5008f1236fdf2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40807 } } }
I20260926 02:59:21.593750 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3caa761ae98049339d5008f1236fdf2c. Latest consensus state: current_term: 1 leader_uuid: "3caa761ae98049339d5008f1236fdf2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3caa761ae98049339d5008f1236fdf2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40807 } } }
I20260926 02:59:21.594174 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:21.594175 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:21.596499 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:21.556834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.0.97.62:40807)
0926 02:59:21.557769 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:21.557786 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:21.559064 (+ 1278us) server_negotiation.cc:407] Connection header received
0926 02:59:21.568829 (+ 9765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:21.568866 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:21.569370 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:21.569800 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:21.572724 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:21.574897 (+ 2173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:21.576554 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:21.577643 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:21.577833 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:21.583839 (+ 6006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:21.583859 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:21.583879 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:21.583935 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:21.588107 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:21.589194 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:21.589202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:21.589209 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:21.589415 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:21.590031 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:21.590042 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:21.590048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:21.590897 (+ 849us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:21.591319 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:21.594247 (+ 2928us) server_negotiation.cc:299] Negotiation successful
0926 02:59:21.594776 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":395,"threads_started":1}
I20260926 02:59:21.597707 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:21.606283 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:21.615890 487 catalog_manager.cc:1347] Generated new cluster ID: 9c5d2513c50a490482c7e420502d3505
I20260926 02:59:21.616001 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:21.634160 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:21.635425 487 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:21.642356 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3caa761ae98049339d5008f1236fdf2c: Generated new TSK 0
I20260926 02:59:21.643350 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:21.686695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40807
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:22.011061 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:22.011358 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:22.011507 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:22.018327 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:22.018427 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:22.018580 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:22.027695 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40807
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:22.028882 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:22.030153 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:22.043694 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:22.043661 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:22.043629 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:22.044795 501 server_base.cc:1034] running on GCE node
I20260926 02:59:22.045563 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:22.046635 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:22.047824 501 hybrid_clock.cc:648] HybridClock initialized: now 1790391562047778 us; error 48 us; skew 500 ppm
I20260926 02:59:22.048130 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:22 dist-test-slave-fccw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:22.051189 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:22.052948 501 webserver.cc:460] Webserver started at http://127.0.97.1:41465/ using document root <none> and password file <none>
I20260926 02:59:22.053661 501 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:22.053802 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:22.054111 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:22.056905 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "78486f384726488f9ddcd7fbe33469a9"
format_stamp: "Formatted at 2026-09-26 02:59:22 on dist-test-slave-fccw"
server_key: "e85d38bd4967b03020192c76e8ddd277"
server_key_iv: "5054ca26a669154e4e7d22579c413708"
server_key_version: "encryptionkey@0"
I20260926 02:59:22.057729 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "78486f384726488f9ddcd7fbe33469a9"
format_stamp: "Formatted at 2026-09-26 02:59:22 on dist-test-slave-fccw"
server_key: "e85d38bd4967b03020192c76e8ddd277"
server_key_iv: "5054ca26a669154e4e7d22579c413708"
server_key_version: "encryptionkey@0"
I20260926 02:59:22.062537 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:59:22.065780 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:22.066993 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:59:22.067194 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "78486f384726488f9ddcd7fbe33469a9"
format_stamp: "Formatted at 2026-09-26 02:59:22 on dist-test-slave-fccw"
server_key: "e85d38bd4967b03020192c76e8ddd277"
server_key_iv: "5054ca26a669154e4e7d22579c413708"
server_key_version: "encryptionkey@0"
I20260926 02:59:22.067380 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:22.083258 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:22.087811 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:22.088096 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:22.089148 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:22.090492 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:22.090591 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:22.090744 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:22.090814 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:22.125168 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39469
I20260926 02:59:22.125190 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39469 every 8 connection(s)
I20260926 02:59:22.126340 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:22.133258 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 501
I20260926 02:59:22.133445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:22.133970 388 external_mini_cluster.cc:1428] Setting key c2771297634d9a1a0a33065cc2f7f85d
Sep 26 02:59:22 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:22.147037 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:22.128951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45387 (local address 127.0.97.62:40807)
0926 02:59:22.129408 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:22.129412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:22.130279 (+ 867us) server_negotiation.cc:407] Connection header received
0926 02:59:22.132018 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:22.132023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:22.132105 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:22.132212 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:22.134055 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.134731 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.136182 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.136543 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.136668 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:22.140107 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:22.140114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:22.140121 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:22.140163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:22.143792 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:22.144427 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:22.144435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:22.144441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:22.144601 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:22.145190 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:22.145198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:22.145204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:22.145571 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:22.145722 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:22.146350 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 02:59:22.146582 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":268,"threads_started":1}
I20260926 02:59:22.147881 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:22.129458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40807 (local address 127.0.97.1:45387)
0926 02:59:22.130112 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:22.130156 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:22.131753 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:22.132827 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:22.132834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:22.133343 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:22.133907 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:22.133921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.134918 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.134922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:22.136039 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:22.136060 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.136757 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.136765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:22.136979 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:22.137672 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:22.137692 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:22.139916 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:22.143923 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:22.143929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:22.143940 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:22.144300 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:22.144852 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:22.144855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:22.144857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:22.145077 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:22.145703 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:22.145712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:22.146007 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 02:59:22.146206 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 02:59:22.146691 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":279,"threads_started":1}
I20260926 02:59:22.149420 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40807
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 02:59:22.149693 629 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:22.150389 629 heartbeater.cc:511] Master 127.0.97.62:40807 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 02:59:22.152562 440 ts_manager.cc:194] Registered new tserver with Master: 78486f384726488f9ddcd7fbe33469a9 (127.0.97.1:39469)
I20260926 02:59:22.154626 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:22.174006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40807
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:22.486012 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:22.486335 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:22.486511 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:22.494112 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:22.494251 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:22.494428 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:22.504233 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40807
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:22.505697 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:22.507081 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:22.519856 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:22.519819 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:22.519886 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:22.520675 635 server_base.cc:1034] running on GCE node
I20260926 02:59:22.521358 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:22.522212 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:22.523427 635 hybrid_clock.cc:648] HybridClock initialized: now 1790391562523392 us; error 51 us; skew 500 ppm
I20260926 02:59:22.523717 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:22 dist-test-slave-fccw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:22.526587 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:22.527814 635 webserver.cc:460] Webserver started at http://127.0.97.2:40259/ using document root <none> and password file <none>
I20260926 02:59:22.528463 635 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:22.528554 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:22.528787 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:22.531591 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "f6e5b7aba11241b9ad1ef75ec711b4e6"
format_stamp: "Formatted at 2026-09-26 02:59:22 on dist-test-slave-fccw"
server_key: "387c014336e51d207d1cb09371593ff9"
server_key_iv: "d1f2fc202f5972fdbf0cce16d34d6ed1"
server_key_version: "encryptionkey@0"
I20260926 02:59:22.532662 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f6e5b7aba11241b9ad1ef75ec711b4e6"
format_stamp: "Formatted at 2026-09-26 02:59:22 on dist-test-slave-fccw"
server_key: "387c014336e51d207d1cb09371593ff9"
server_key_iv: "d1f2fc202f5972fdbf0cce16d34d6ed1"
server_key_version: "encryptionkey@0"
I20260926 02:59:22.537453 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 02:59:22.540462 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:22.541714 635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:22.541841 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "f6e5b7aba11241b9ad1ef75ec711b4e6"
format_stamp: "Formatted at 2026-09-26 02:59:22 on dist-test-slave-fccw"
server_key: "387c014336e51d207d1cb09371593ff9"
server_key_iv: "d1f2fc202f5972fdbf0cce16d34d6ed1"
server_key_version: "encryptionkey@0"
I20260926 02:59:22.542013 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:22.563000 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:22.568166 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:22.568534 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:22.578063 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:22.579372 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:22.579484 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:22.579675 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:22.579746 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:22.612516 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42703
I20260926 02:59:22.612535 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42703 every 8 connection(s)
I20260926 02:59:22.613819 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:22.620004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 635
I20260926 02:59:22.620189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:22.620710 388 external_mini_cluster.cc:1428] Setting key 12562b691ccf370a57369ab95b7315d3
Sep 26 02:59:22 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:22.633962 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:22.616540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48465 (local address 127.0.97.62:40807)
0926 02:59:22.616739 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:22.616745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:22.617906 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 02:59:22.619526 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:22.619530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:22.619590 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:22.619675 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:22.621623 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.622357 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.623865 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.624232 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.624358 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:22.628013 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:22.628017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:22.628020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:22.628059 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:22.630509 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:22.631409 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:22.631413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:22.631415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:22.631526 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:22.632399 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:22.632409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:22.632411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:22.632639 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:22.632740 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:22.633597 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 02:59:22.633744 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 02:59:22.635423 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:22.617112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40807 (local address 127.0.97.2:48465)
0926 02:59:22.617748 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:22.617783 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:22.619291 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:22.619943 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:22.619956 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:22.620652 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:22.621447 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:22.621465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.622503 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.622508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:22.623725 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:22.623735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:22.624450 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:22.624458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:22.624655 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:22.625356 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:22.625381 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:22.627775 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:22.630635 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:22.630646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:22.630662 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:22.631289 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:22.631631 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:22.631638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:22.631645 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:22.632280 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:22.632760 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:22.632771 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:22.633230 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 02:59:22.633539 (+ 309us) client_negotiation.cc:240] Negotiation successful
0926 02:59:22.634084 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 02:59:22.636816 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40807
I20260926 02:59:22.637144 763 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:22.637861 763 heartbeater.cc:511] Master 127.0.97.62:40807 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 02:59:22.639231 440 ts_manager.cc:194] Registered new tserver with Master: f6e5b7aba11241b9ad1ef75ec711b4e6 (127.0.97.2:42703)
I20260926 02:59:22.640014 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:22.658354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40807
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:22.971843 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:22.972105 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:22.972267 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:22.979079 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:22.979173 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:22.979339 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:22.988550 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40807
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:22.990131 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:22.991541 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:23.004509 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:23.004593 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:23.004520 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:23.005740 768 server_base.cc:1034] running on GCE node
I20260926 02:59:23.006319 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:23.007208 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:23.008404 768 hybrid_clock.cc:648] HybridClock initialized: now 1790391563008360 us; error 39 us; skew 500 ppm
I20260926 02:59:23.008648 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:23.011658 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:23.012984 768 webserver.cc:460] Webserver started at http://127.0.97.3:42163/ using document root <none> and password file <none>
I20260926 02:59:23.013724 768 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:23.013860 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:23.014180 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:23.016929 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "2dd71d3e659f419ab68b37a0d94848d4"
format_stamp: "Formatted at 2026-09-26 02:59:23 on dist-test-slave-fccw"
server_key: "58fb143fdb12b2d1ee1b77c2dba89835"
server_key_iv: "724b0023cb6873b194af401bcc4340bf"
server_key_version: "encryptionkey@0"
I20260926 02:59:23.017724 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2dd71d3e659f419ab68b37a0d94848d4"
format_stamp: "Formatted at 2026-09-26 02:59:23 on dist-test-slave-fccw"
server_key: "58fb143fdb12b2d1ee1b77c2dba89835"
server_key_iv: "724b0023cb6873b194af401bcc4340bf"
server_key_version: "encryptionkey@0"
I20260926 02:59:23.022429 768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:59:23.025517 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:23.026676 768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 02:59:23.027287 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "2dd71d3e659f419ab68b37a0d94848d4"
format_stamp: "Formatted at 2026-09-26 02:59:23 on dist-test-slave-fccw"
server_key: "58fb143fdb12b2d1ee1b77c2dba89835"
server_key_iv: "724b0023cb6873b194af401bcc4340bf"
server_key_version: "encryptionkey@0"
I20260926 02:59:23.027486 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:23.061697 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:23.065382 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:23.065663 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:23.066548 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:23.067778 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:23.067879 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:23.068037 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:23.068110 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:23.107306 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39679
I20260926 02:59:23.107352 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39679 every 8 connection(s)
I20260926 02:59:23.108544 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:23.113330 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 768
I20260926 02:59:23.113521 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:23.113927 388 external_mini_cluster.cc:1428] Setting key 72d13e15f13898fbc4315de8f182b21f
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:23.128584 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:23.110984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49557 (local address 127.0.97.62:40807)
0926 02:59:23.111172 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:23.111178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:23.112275 (+ 1097us) server_negotiation.cc:407] Connection header received
0926 02:59:23.114137 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:23.114143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:23.114224 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:23.114338 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:23.115810 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.116769 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.118501 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.118885 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.118994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:23.122224 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:23.122229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:23.122232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:23.122262 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:23.125533 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.126200 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.126206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.126210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.126347 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.126860 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.126865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.126867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.127201 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:23.127338 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:23.127953 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 02:59:23.128167 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:59:23.129633 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:23.111478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40807 (local address 127.0.97.3:49557)
0926 02:59:23.112121 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:23.112161 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:23.113859 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:23.114534 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:23.114541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:23.115097 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:23.115678 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:23.115694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.116963 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.116968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:23.118346 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:23.118356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.119069 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.119074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:23.119281 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:23.119886 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:23.119913 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:23.122045 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:23.125653 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:23.125660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:23.125670 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:23.126065 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:23.126558 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:23.126561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:23.126564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:23.126732 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:23.127311 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:23.127315 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:23.127574 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 02:59:23.127824 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:59:23.128294 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":259,"threads_started":1}
I20260926 02:59:23.130957 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40807
I20260926 02:59:23.131239 896 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:23.131904 896 heartbeater.cc:511] Master 127.0.97.62:40807 requested a full tablet report, sending...
I20260926 02:59:23.133133 440 ts_manager.cc:194] Registered new tserver with Master: 2dd71d3e659f419ab68b37a0d94848d4 (127.0.97.3:39679)
I20260926 02:59:23.133909 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49557
I20260926 02:59:23.141602 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:59:23.158079 629 heartbeater.cc:503] Master 127.0.97.62:40807 was elected leader, sending a full tablet report...
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:23.185235 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:23.169666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.97.1:39469)
0926 02:59:23.170328 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:23.170339 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:23.170417 (+ 78us) server_negotiation.cc:407] Connection header received
0926 02:59:23.170631 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:23.170638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:23.170859 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:23.171029 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:23.172784 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.173848 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.175112 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.175659 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.175794 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:23.179634 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:23.179645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:23.179659 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:23.179693 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:23.182487 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.183167 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.183174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.183176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.183293 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.183671 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.183678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.183681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.184027 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:23.184313 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:23.184618 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 02:59:23.184874 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":302,"threads_started":1}
W20260926 02:59:23.186379 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40792
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:23.227602 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:23.215568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.0.97.1:39469)
0926 02:59:23.215802 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:23.215809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:23.215924 (+ 115us) server_negotiation.cc:407] Connection header received
0926 02:59:23.216072 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:23.216075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:23.216136 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:23.216219 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:23.217525 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.218190 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.219293 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.219665 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.219750 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:23.223303 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:23.223307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:23.223311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:23.223342 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:23.225649 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.226180 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.226183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.226185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.226286 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.226586 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.226588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.226590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.226794 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:23.226987 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:23.227213 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 02:59:23.227370 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 02:59:23.229156 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 501
I20260926 02:59:23.240688 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:23.230063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.0.97.1:39469)
0926 02:59:23.230219 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:23.230257 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:23.230471 (+ 214us) server_negotiation.cc:407] Connection header received
0926 02:59:23.230573 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:23.230576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:23.230640 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:23.230710 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:23.231774 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.232484 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.233660 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.234063 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.234149 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:23.236115 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:23.236119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:23.236122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:23.236146 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:23.238702 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.239202 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.239205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.239207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.239311 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.239662 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.239667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.239671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.239909 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:23.240063 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:23.240282 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 02:59:23.240435 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 02:59:23.241406 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:23.323200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 501
I20260926 02:59:23.339447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 635
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 02:59:23.354164 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:23.340345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42618 (local address 127.0.97.2:42703)
0926 02:59:23.340835 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:23.340842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:23.340866 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:59:23.340930 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:23.340936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:23.341200 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:23.341328 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:23.342407 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.343420 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.344559 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.345261 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.345356 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:23.349110 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:23.349120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:23.349134 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:23.349164 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:23.351917 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.352464 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.352471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.352473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.352582 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.352896 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.352899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.352901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.353259 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:23.353479 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:23.353712 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 02:59:23.353890 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":228,"threads_started":1}
I20260926 02:59:23.355379 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:23.437117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 635
I20260926 02:59:23.459066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 768
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 02:59:23.473937 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:23.460093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.97.3:39679)
0926 02:59:23.460700 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:23.460706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:23.460733 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:59:23.460792 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:23.460799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:23.461054 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:23.461201 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:23.462289 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.463262 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.464395 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:23.464900 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:23.465050 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:23.469082 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:23.469093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:23.469108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:23.469140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:23.471743 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.472288 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.472294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.472296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.472412 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:23.472702 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:23.472705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:23.472707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:23.473043 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:23.473270 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:23.473498 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 02:59:23.473675 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":313,"threads_started":1}
I20260926 02:59:23.475107 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:23.556628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 768
I20260926 02:59:23.572686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 408
I20260926 02:59:23.573845 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:23.673249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 408
2026-09-26T02:59:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5747 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[917](info): set up 2 sockets
Sep 26 02:59:23 dist-test-slave-fccw krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 02:59:24.780941 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:24 dist-test-slave-fccw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:24.943739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33919
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:25.249015 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:25.249339 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:25.249500 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:25.256525 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:25.256669 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:25.256776 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:25.256883 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:25.256968 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:25.266594 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:25.267832 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:25.269208 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:25.280556 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:25.280663 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:25.280624 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:25.281298 930 server_base.cc:1034] running on GCE node
I20260926 02:59:25.282143 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:25.283432 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:25.284651 930 hybrid_clock.cc:648] HybridClock initialized: now 1790391565284611 us; error 52 us; skew 500 ppm
I20260926 02:59:25.284974 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:25 dist-test-slave-fccw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:25.287937 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:25.289386 930 webserver.cc:460] Webserver started at http://127.0.97.62:44699/ using document root <none> and password file <none>
I20260926 02:59:25.290040 930 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:25.290179 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:25.290484 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:25.293335 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "e7050b467c084bc2b7c092f5259faf41"
format_stamp: "Formatted at 2026-09-26 02:59:25 on dist-test-slave-fccw"
server_key: "5759c5155d37dcf76357288bd2ebd655"
server_key_iv: "624338795e83ece0f1bacdc38caca3d2"
server_key_version: "encryptionkey@0"
I20260926 02:59:25.294090 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "e7050b467c084bc2b7c092f5259faf41"
format_stamp: "Formatted at 2026-09-26 02:59:25 on dist-test-slave-fccw"
server_key: "5759c5155d37dcf76357288bd2ebd655"
server_key_iv: "624338795e83ece0f1bacdc38caca3d2"
server_key_version: "encryptionkey@0"
I20260926 02:59:25.298861 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:59:25.301988 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:25.303164 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 02:59:25.303341 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "e7050b467c084bc2b7c092f5259faf41"
format_stamp: "Formatted at 2026-09-26 02:59:25 on dist-test-slave-fccw"
server_key: "5759c5155d37dcf76357288bd2ebd655"
server_key_iv: "624338795e83ece0f1bacdc38caca3d2"
server_key_version: "encryptionkey@0"
I20260926 02:59:25.303516 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:25.337272 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:25.340945 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:25.341243 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:25.358562 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33919
I20260926 02:59:25.358597 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33919 every 8 connection(s)
I20260926 02:59:25.359939 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:25.363565 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:25.367813 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 930
I20260926 02:59:25.368034 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:25.368813 388 external_mini_cluster.cc:1428] Setting key 7d73ef3f771df6dd497d02a1f8c1fc7f
I20260926 02:59:25.374699 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41: Bootstrap starting.
I20260926 02:59:25.377698 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:25.378647 997 log.cc:824] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:25.381939 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41: No bootstrap required, opened a new log
Sep 26 02:59:25 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:25.389403 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7050b467c084bc2b7c092f5259faf41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33919 } }
I20260926 02:59:25.389832 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:25.389959 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7050b467c084bc2b7c092f5259faf41, State: Initialized, Role: FOLLOWER
I20260926 02:59:25.390591 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7050b467c084bc2b7c092f5259faf41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33919 } }
I20260926 02:59:25.390869 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:25.370860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.0.97.62:33919)
0926 02:59:25.371656 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:25.371667 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:25.371705 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:59:25.373510 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:25.373531 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:25.373893 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:25.374251 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:25.375696 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:25.376984 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:25.378308 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:25.379035 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:25.379257 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:25.383626 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:25.383633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:25.383649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:25.383685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:25.386811 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:25.387474 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:25.387488 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:25.387493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:25.387655 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:25.388010 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:25.388015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:25.388020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:25.388508 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:25.388834 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:25.389321 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 02:59:25.389721 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":375,"threads_started":1}
I20260926 02:59:25.390887 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:25.391054 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:25.391256 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:25.393615 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7050b467c084bc2b7c092f5259faf41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33919 } }
I20260926 02:59:25.394105 997 leader_election.cc:302] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7050b467c084bc2b7c092f5259faf41; no voters:
I20260926 02:59:25.394642 997 leader_election.cc:288] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:25.394760 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:25.395390 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [term 1 LEADER]: Becoming Leader. State: Replica: e7050b467c084bc2b7c092f5259faf41, State: Running, Role: LEADER
I20260926 02:59:25.395896 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7050b467c084bc2b7c092f5259faf41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33919 } }
I20260926 02:59:25.396539 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:25.398422 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7050b467c084bc2b7c092f5259faf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7050b467c084bc2b7c092f5259faf41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33919 } } }
I20260926 02:59:25.398729 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:25.399487 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:25.399737 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7050b467c084bc2b7c092f5259faf41. Latest consensus state: current_term: 1 leader_uuid: "e7050b467c084bc2b7c092f5259faf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7050b467c084bc2b7c092f5259faf41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33919 } } }
I20260926 02:59:25.400127 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:25.404682 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:25.414098 1007 catalog_manager.cc:1347] Generated new cluster ID: 0ba0fb2d3dab497ba6b35ca03c12ec08
I20260926 02:59:25.414207 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:25.435425 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:25.436628 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:25.445729 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7050b467c084bc2b7c092f5259faf41: Generated new TSK 0
I20260926 02:59:25.446565 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:25.502435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33919
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:25.819319 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:25.819561 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:25.819648 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:25.826524 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:25.826627 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:25.826772 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:25.837116 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:25.838285 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:25.839584 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:25.852329 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:25.852417 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:25.852371 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:25.854184 1023 server_base.cc:1034] running on GCE node
I20260926 02:59:25.854717 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:25.855571 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:25.856752 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790391565856720 us; error 30 us; skew 500 ppm
I20260926 02:59:25.857098 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:25 dist-test-slave-fccw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:25.859913 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:25.861395 1023 webserver.cc:460] Webserver started at http://127.0.97.1:41743/ using document root <none> and password file <none>
I20260926 02:59:25.862049 1023 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:25.862185 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:25.862491 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:25.865204 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "66be2acee53b4ac9a40a9112844bcacd"
format_stamp: "Formatted at 2026-09-26 02:59:25 on dist-test-slave-fccw"
server_key: "a65e8cf372526e615447e93b1388eba6"
server_key_iv: "469820678d1911c07564aee73503ffa2"
server_key_version: "encryptionkey@0"
I20260926 02:59:25.865940 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "66be2acee53b4ac9a40a9112844bcacd"
format_stamp: "Formatted at 2026-09-26 02:59:25 on dist-test-slave-fccw"
server_key: "a65e8cf372526e615447e93b1388eba6"
server_key_iv: "469820678d1911c07564aee73503ffa2"
server_key_version: "encryptionkey@0"
I20260926 02:59:25.870718 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:59:25.873893 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:25.875063 1023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:25.875228 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "66be2acee53b4ac9a40a9112844bcacd"
format_stamp: "Formatted at 2026-09-26 02:59:25 on dist-test-slave-fccw"
server_key: "a65e8cf372526e615447e93b1388eba6"
server_key_iv: "469820678d1911c07564aee73503ffa2"
server_key_version: "encryptionkey@0"
I20260926 02:59:25.875428 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:25.887156 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:25.890991 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:25.891285 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:25.892165 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:25.893364 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:25.893457 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:25.893594 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:25.893659 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:25.923745 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41559
I20260926 02:59:25.923789 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41559 every 8 connection(s)
I20260926 02:59:25.924949 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:25.930119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 1023
I20260926 02:59:25.930307 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:25.930971 388 external_mini_cluster.cc:1428] Setting key 8c74a6d95878444b7e6dc31139a2c18c
Sep 26 02:59:25 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391565, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:25.945076 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:25.927390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53175 (local address 127.0.97.62:33919)
0926 02:59:25.927820 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:25.927826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:25.928751 (+ 925us) server_negotiation.cc:407] Connection header received
0926 02:59:25.930408 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:25.930413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:25.930494 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:25.930617 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:25.932139 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:25.933143 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:25.935567 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:25.935927 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:25.936007 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:25.939668 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:25.939673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:25.939676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:25.939703 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:25.942175 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:25.942885 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:25.942892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:25.942895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:25.943026 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:25.943437 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:25.943444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:25.943449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:25.943702 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:25.943856 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:25.944463 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 02:59:25.944676 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":219,"threads_started":1}
I20260926 02:59:25.946022 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:25.927959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33919 (local address 127.0.97.1:53175)
0926 02:59:25.928600 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:25.928641 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:25.930124 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:25.930799 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:25.930806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:25.931334 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:25.931985 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:25.932003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:25.933277 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:25.933283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:25.935436 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:25.935453 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:25.936122 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:25.936130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:25.936442 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:25.937137 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:25.937167 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:25.939461 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:25.942332 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:25.942340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:25.942356 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:25.942731 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:25.943139 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:25.943144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:25.943147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:25.943319 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:25.943828 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:25.943836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:25.944133 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 02:59:25.944383 (+ 250us) client_negotiation.cc:240] Negotiation successful
0926 02:59:25.944864 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 02:59:25.947492 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33919
I20260926 02:59:25.947804 1151 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:25.948474 1151 heartbeater.cc:511] Master 127.0.97.62:33919 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 02:59:25.950510 962 ts_manager.cc:194] Registered new tserver with Master: 66be2acee53b4ac9a40a9112844bcacd (127.0.97.1:41559)
I20260926 02:59:25.951891 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:25.968106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33919
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:26.279158 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:26.279397 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:26.279582 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:26.286422 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:26.286520 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:26.286629 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:26.295672 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:26.296825 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:26.298127 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:26.310665 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:26.310751 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:26.310715 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:26.311898 1157 server_base.cc:1034] running on GCE node
I20260926 02:59:26.312428 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:26.313373 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:26.314580 1157 hybrid_clock.cc:648] HybridClock initialized: now 1790391566314524 us; error 49 us; skew 500 ppm
I20260926 02:59:26.314903 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:26 dist-test-slave-fccw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:26.317777 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:26.319111 1157 webserver.cc:460] Webserver started at http://127.0.97.2:41931/ using document root <none> and password file <none>
I20260926 02:59:26.319731 1157 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:26.319866 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:26.320152 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:26.322959 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "25ad6e8240b240ff87e21c5ce911c007"
format_stamp: "Formatted at 2026-09-26 02:59:26 on dist-test-slave-fccw"
server_key: "080603c23d581f9649c3209417da6dfa"
server_key_iv: "c7cfa81f8b2a59f46a2546aeb8896e70"
server_key_version: "encryptionkey@0"
I20260926 02:59:26.323679 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "25ad6e8240b240ff87e21c5ce911c007"
format_stamp: "Formatted at 2026-09-26 02:59:26 on dist-test-slave-fccw"
server_key: "080603c23d581f9649c3209417da6dfa"
server_key_iv: "c7cfa81f8b2a59f46a2546aeb8896e70"
server_key_version: "encryptionkey@0"
I20260926 02:59:26.328415 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 02:59:26.331523 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:26.332712 1157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 02:59:26.332885 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "25ad6e8240b240ff87e21c5ce911c007"
format_stamp: "Formatted at 2026-09-26 02:59:26 on dist-test-slave-fccw"
server_key: "080603c23d581f9649c3209417da6dfa"
server_key_iv: "c7cfa81f8b2a59f46a2546aeb8896e70"
server_key_version: "encryptionkey@0"
I20260926 02:59:26.333120 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:26.354468 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:26.358301 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:26.358572 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:26.359488 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:26.360749 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:26.360850 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:26.360989 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:26.361097 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 02:59:26.395188 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39625
I20260926 02:59:26.395231 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39625 every 8 connection(s)
I20260926 02:59:26.396484 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:26.402376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 1157
I20260926 02:59:26.402586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:26.403123 388 external_mini_cluster.cc:1428] Setting key 222c29e8177235bc63e90abe3df047d0
Sep 26 02:59:26 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:26.418247 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.399112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45915 (local address 127.0.97.62:33919)
0926 02:59:26.399288 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:26.399292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:26.400413 (+ 1121us) server_negotiation.cc:407] Connection header received
0926 02:59:26.402274 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:26.402282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:26.402363 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:26.402449 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:26.404287 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.405330 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.406881 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.407348 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.407474 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.411877 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:26.411882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:26.411885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:26.411914 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:26.415175 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:26.415975 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:26.415983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:26.415988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:26.416152 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:26.416590 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:26.416595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:26.416598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:26.416855 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:26.416959 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:26.417865 (+ 906us) server_negotiation.cc:299] Negotiation successful
0926 02:59:26.418018 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":71}
I20260926 02:59:26.419768 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.399596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33919 (local address 127.0.97.2:45915)
0926 02:59:26.400247 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:26.400291 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:26.401936 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:26.402769 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:26.402782 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:26.403467 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:26.404139 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.404155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.405482 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.405487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.406746 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.406762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.407548 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.407556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.407874 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.408617 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:26.408646 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:26.411684 (+ 3038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:26.415450 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:26.415458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:26.415480 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:26.415863 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:26.416263 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:26.416266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:26.416268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:26.416484 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:26.416982 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:26.417016 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:26.417485 (+ 469us) client_negotiation.cc:769] Sending connection context
0926 02:59:26.417815 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 02:59:26.418357 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 02:59:26.421134 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33919
I20260926 02:59:26.421435 1285 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:26.422096 1285 heartbeater.cc:511] Master 127.0.97.62:33919 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 02:59:26.423462 962 ts_manager.cc:194] Registered new tserver with Master: 25ad6e8240b240ff87e21c5ce911c007 (127.0.97.2:39625)
I20260926 02:59:26.424248 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:26.442118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33919
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:26.745219 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:26.745479 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:26.745626 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:26.752488 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:26.752605 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:26.752764 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:26.762089 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33919
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:26.763322 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:26.764597 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:26.777259 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:26.777330 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:26.777199 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:26.778007 1290 server_base.cc:1034] running on GCE node
I20260926 02:59:26.778558 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:26.779500 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:26.780735 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790391566780656 us; error 68 us; skew 500 ppm
I20260926 02:59:26.781081 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:26 dist-test-slave-fccw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:26.787060 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:26.788378 1290 webserver.cc:460] Webserver started at http://127.0.97.3:34385/ using document root <none> and password file <none>
I20260926 02:59:26.789083 1290 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:26.789224 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:26.789512 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:26.792220 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "cecd023433c447f596909060be5254ef"
format_stamp: "Formatted at 2026-09-26 02:59:26 on dist-test-slave-fccw"
server_key: "f3283db90e6be384b8e39e6ffa2693e7"
server_key_iv: "818e8c9cf3de6214946d2b8e0c313cc1"
server_key_version: "encryptionkey@0"
I20260926 02:59:26.792965 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cecd023433c447f596909060be5254ef"
format_stamp: "Formatted at 2026-09-26 02:59:26 on dist-test-slave-fccw"
server_key: "f3283db90e6be384b8e39e6ffa2693e7"
server_key_iv: "818e8c9cf3de6214946d2b8e0c313cc1"
server_key_version: "encryptionkey@0"
I20260926 02:59:26.797876 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:59:26.801091 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:26.802352 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 02:59:26.802508 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "cecd023433c447f596909060be5254ef"
format_stamp: "Formatted at 2026-09-26 02:59:26 on dist-test-slave-fccw"
server_key: "f3283db90e6be384b8e39e6ffa2693e7"
server_key_iv: "818e8c9cf3de6214946d2b8e0c313cc1"
server_key_version: "encryptionkey@0"
I20260926 02:59:26.802690 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:26.820179 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:26.824074 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:26.824375 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:26.825363 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:26.826817 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:26.826885 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:26.827021 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:26.827093 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:26.860471 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37803
I20260926 02:59:26.860512 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37803 every 8 connection(s)
I20260926 02:59:26.861754 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:26.866653 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 1290
I20260926 02:59:26.866849 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:26.867379 388 external_mini_cluster.cc:1428] Setting key d90217932441c9ae92c9b445d00cb9cd
Sep 26 02:59:26 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:26.881995 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.864203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57477 (local address 127.0.97.62:33919)
0926 02:59:26.864417 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:26.864423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:26.865646 (+ 1223us) server_negotiation.cc:407] Connection header received
0926 02:59:26.867955 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:26.867961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:26.868049 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:26.868161 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:26.870046 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.870817 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.872674 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.873113 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.873201 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.876411 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:26.876415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:26.876418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:26.876446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:26.878837 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:26.879531 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:26.879535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:26.879537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:26.879643 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:26.879987 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:26.879990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:26.879992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:26.880293 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:26.880398 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:26.881325 (+ 927us) server_negotiation.cc:299] Negotiation successful
0926 02:59:26.881553 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 02:59:26.883086 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.864717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33919 (local address 127.0.97.3:57477)
0926 02:59:26.865483 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:26.865528 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:26.867641 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:26.868431 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:26.868440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:26.869213 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:26.869835 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.869852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.870942 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.870946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.872562 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.872573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.873280 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.873285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.873477 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.874160 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:26.874185 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:26.876216 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:26.878953 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:26.878960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:26.878972 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:26.879429 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:26.879732 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:26.879735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:26.879737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:26.879902 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:26.880414 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:26.880422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:26.880851 (+ 429us) client_negotiation.cc:769] Sending connection context
0926 02:59:26.881190 (+ 339us) client_negotiation.cc:240] Negotiation successful
0926 02:59:26.881711 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
I20260926 02:59:26.884353 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33919
I20260926 02:59:26.884642 1418 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:26.885388 1418 heartbeater.cc:511] Master 127.0.97.62:33919 requested a full tablet report, sending...
I20260926 02:59:26.886732 962 ts_manager.cc:194] Registered new tserver with Master: cecd023433c447f596909060be5254ef (127.0.97.3:37803)
I20260926 02:59:26.887481 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57477
I20260926 02:59:26.894871 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:59:26.916471 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.900962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.0.97.62:33919)
0926 02:59:26.901201 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:26.901209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:26.901641 (+ 432us) server_negotiation.cc:407] Connection header received
0926 02:59:26.901857 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:26.901861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:26.901920 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:26.902022 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:26.904130 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.904846 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.906593 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.907437 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.907581 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.910145 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:26.910153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:26.910158 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:26.910200 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:26.913692 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:26.914347 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:26.914352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:26.914358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:26.914508 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:26.914849 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:26.914855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:26.914860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:26.915290 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:26.915543 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:26.915843 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 02:59:26.916079 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 02:59:26.923647 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:59:26.934703 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:26.955004 1151 heartbeater.cc:503] Master 127.0.97.62:33919 was elected leader, sending a full tablet report...
I20260926 02:59:26.960011 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.948798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39625 (local address 127.0.0.1:59306)
0926 02:59:26.949501 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:26.949544 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:26.949709 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:26.951123 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:26.951133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:26.951198 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:26.951589 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.951604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.953813 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.953831 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.956576 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.956600 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.958845 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.958854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.959105 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.959193 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 02:59:26.959314 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 02:59:26.959533 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":327,"threads_started":1}
I20260926 02:59:26.961609 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.948809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.0.97.2:39625)
0926 02:59:26.950213 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:26.950221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:26.950251 (+ 30us) server_negotiation.cc:407] Connection header received
0926 02:59:26.950348 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:26.950355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:26.950622 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:26.950813 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:26.951801 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.953649 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.956813 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.958602 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.960633 (+ 2031us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.960898 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:26.960972 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 02:59:26.961203 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":1257,"threads_started":1}
I20260926 02:59:26.970890 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 26707c4c1bcf4ed49aa1197455fd68a6 (DEFAULT_TABLE table=test-table [id=82956eaef39f4c469806e3bb03531cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:26.972517 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26707c4c1bcf4ed49aa1197455fd68a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:26.974817 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.949343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37803 (local address 127.0.0.1:35702)
0926 02:59:26.960280 (+ 10937us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:26.960302 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:26.960435 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:26.967230 (+ 6795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:26.967234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:26.967256 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:26.967439 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.967451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.969808 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.969812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.971619 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.971631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.974097 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.974103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.974282 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.974328 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 02:59:26.974394 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 02:59:26.974523 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10820,"thread_start_us":16845,"threads_started":1}
I20260926 02:59:26.975836 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.949889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41559 (local address 127.0.0.1:43094)
0926 02:59:26.965731 (+ 15842us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:26.965753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:26.965884 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:26.968919 (+ 3035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:26.968924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:26.968945 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:26.969280 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.969290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.971314 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.971318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.973337 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:26.973350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.975108 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.975114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:26.975292 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.975322 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:59:26.975385 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 02:59:26.975509 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15708,"thread_start_us":15544,"threads_started":1}
I20260926 02:59:26.976042 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.950030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.0.97.3:37803)
0926 02:59:26.966636 (+ 16606us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:26.966651 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:26.966707 (+ 56us) server_negotiation.cc:407] Connection header received
0926 02:59:26.966796 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:26.966804 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:26.967043 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:26.967251 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:26.967627 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.969646 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.972545 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.973930 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.975221 (+ 1291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.975469 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:26.975537 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 02:59:26.975696 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16456,"thread_start_us":16296,"threads_started":1}
I20260926 02:59:26.976763 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:26.950334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.0.97.1:41559)
0926 02:59:26.968330 (+ 17996us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:26.968338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:26.968365 (+ 27us) server_negotiation.cc:407] Connection header received
0926 02:59:26.968457 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:26.968465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:26.968680 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:26.969464 (+ 784us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:26.969663 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.971177 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.973529 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:26.974893 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:26.976030 (+ 1137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:26.976269 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:26.976355 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 02:59:26.976495 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17820,"threads_started":1}
I20260926 02:59:26.979561 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 26707c4c1bcf4ed49aa1197455fd68a6 (DEFAULT_TABLE table=test-table [id=82956eaef39f4c469806e3bb03531cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:26.981119 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26707c4c1bcf4ed49aa1197455fd68a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:26.981194 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 26707c4c1bcf4ed49aa1197455fd68a6 (DEFAULT_TABLE table=test-table [id=82956eaef39f4c469806e3bb03531cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:26.982508 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26707c4c1bcf4ed49aa1197455fd68a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:26.995447 1436 tablet_bootstrap.cc:492] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007: Bootstrap starting.
I20260926 02:59:26.997026 1437 tablet_bootstrap.cc:492] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef: Bootstrap starting.
I20260926 02:59:27.000039 1437 tablet_bootstrap.cc:654] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:27.001096 1436 tablet_bootstrap.cc:654] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:27.001379 1438 tablet_bootstrap.cc:492] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd: Bootstrap starting.
I20260926 02:59:27.001780 1437 log.cc:824] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:27.002264 1436 log.cc:824] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:27.004706 1438 tablet_bootstrap.cc:654] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:27.004827 1437 tablet_bootstrap.cc:492] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef: No bootstrap required, opened a new log
I20260926 02:59:27.005124 1437 ts_tablet_manager.cc:1397] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260926 02:59:27.005273 1436 tablet_bootstrap.cc:492] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007: No bootstrap required, opened a new log
I20260926 02:59:27.005535 1436 ts_tablet_manager.cc:1397] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 02:59:27.006049 1438 log.cc:824] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:27.011116 1438 tablet_bootstrap.cc:492] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd: No bootstrap required, opened a new log
I20260926 02:59:27.011369 1438 ts_tablet_manager.cc:1397] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260926 02:59:27.016582 1436 raft_consensus.cc:348] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.016582 1437 raft_consensus.cc:348] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.017040 1436 raft_consensus.cc:374] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:27.017046 1437 raft_consensus.cc:374] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:27.017134 1436 raft_consensus.cc:729] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ad6e8240b240ff87e21c5ce911c007, State: Initialized, Role: FOLLOWER
I20260926 02:59:27.017141 1437 raft_consensus.cc:729] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cecd023433c447f596909060be5254ef, State: Initialized, Role: FOLLOWER
I20260926 02:59:27.017836 1437 consensus_queue.cc:260] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.017836 1436 consensus_queue.cc:260] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.019313 1418 heartbeater.cc:503] Master 127.0.97.62:33919 was elected leader, sending a full tablet report...
I20260926 02:59:27.019294 1285 heartbeater.cc:503] Master 127.0.97.62:33919 was elected leader, sending a full tablet report...
I20260926 02:59:27.019584 1437 ts_tablet_manager.cc:1428] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 02:59:27.019927 1436 ts_tablet_manager.cc:1428] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 02:59:27.019064 1438 raft_consensus.cc:348] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.020251 1438 raft_consensus.cc:374] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:27.020352 1438 raft_consensus.cc:729] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66be2acee53b4ac9a40a9112844bcacd, State: Initialized, Role: FOLLOWER
I20260926 02:59:27.020911 1438 consensus_queue.cc:260] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.022562 1438 ts_tablet_manager.cc:1428] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260926 02:59:27.072211 1444 raft_consensus.cc:482] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:27.072506 1444 raft_consensus.cc:504] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.074316 1444 leader_election.cc:288] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25ad6e8240b240ff87e21c5ce911c007 (127.0.97.2:39625), cecd023433c447f596909060be5254ef (127.0.97.3:37803)
I20260926 02:59:27.076962 1442 raft_consensus.cc:482] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:27.077281 1442 raft_consensus.cc:504] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.083705 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.074725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42301 (local address 127.0.97.2:39625)
0926 02:59:27.074971 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.074978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.076111 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 02:59:27.076319 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.076327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.076426 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.076526 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:27.077301 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.078392 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.081158 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.082435 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.082537 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.082630 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.083315 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.083441 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 02:59:27.083704 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.074725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49517 (local address 127.0.97.3:37803)
0926 02:59:27.074971 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.074978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.076272 (+ 1294us) server_negotiation.cc:407] Connection header received
0926 02:59:27.076421 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.076426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.076505 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.076588 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:27.077331 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.078282 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.081156 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.082435 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.082537 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.082636 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.083310 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.083441 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 02:59:27.084651 1442 leader_election.cc:288] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25ad6e8240b240ff87e21c5ce911c007 (127.0.97.2:39625), 66be2acee53b4ac9a40a9112844bcacd (127.0.97.1:41559)
I20260926 02:59:27.084878 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.075125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37803 (local address 127.0.97.1:49517)
0926 02:59:27.076158 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:27.076172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:27.076290 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:27.076604 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:27.076608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:27.076625 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:27.077133 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.077141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.078411 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.078417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.080956 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.080970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.082843 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.082851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.083105 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.083157 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:59:27.084165 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0926 02:59:27.084399 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":178,"thread_start_us":414,"threads_started":1}
I20260926 02:59:27.085969 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26707c4c1bcf4ed49aa1197455fd68a6" candidate_uuid: "66be2acee53b4ac9a40a9112844bcacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cecd023433c447f596909060be5254ef" is_pre_election: true
I20260926 02:59:27.086522 1373 raft_consensus.cc:2454] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66be2acee53b4ac9a40a9112844bcacd in term 0.
I20260926 02:59:27.087211 1041 leader_election.cc:302] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66be2acee53b4ac9a40a9112844bcacd, cecd023433c447f596909060be5254ef; no voters:
I20260926 02:59:27.087715 1444 raft_consensus.cc:2781] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:27.087934 1444 raft_consensus.cc:482] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:27.088066 1444 raft_consensus.cc:3037] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:27.091059 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.079802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39625 (local address 127.0.97.3:46809)
0926 02:59:27.080644 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:27.080706 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:27.080907 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:27.084484 (+ 3577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:27.084488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:27.084528 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:27.084702 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.084712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.086945 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.086950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.088811 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.088824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.090334 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.090340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.090570 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.090606 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:59:27.090668 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 02:59:27.090792 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260926 02:59:27.091434 1444 raft_consensus.cc:504] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.092833 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26707c4c1bcf4ed49aa1197455fd68a6" candidate_uuid: "66be2acee53b4ac9a40a9112844bcacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cecd023433c447f596909060be5254ef"
I20260926 02:59:27.093092 1373 raft_consensus.cc:3037] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:27.096534 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.079884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46809 (local address 127.0.97.2:39625)
0926 02:59:27.083991 (+ 4107us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.083995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.084010 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:59:27.084057 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.084061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.084127 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.084213 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:27.084886 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.086022 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.089018 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.090175 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.096017 (+ 5842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.096122 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.096167 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.096284 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4010,"threads_started":1}
I20260926 02:59:27.097018 1373 raft_consensus.cc:2454] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66be2acee53b4ac9a40a9112844bcacd in term 1.
I20260926 02:59:27.097436 1041 leader_election.cc:302] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66be2acee53b4ac9a40a9112844bcacd, cecd023433c447f596909060be5254ef; no voters:
I20260926 02:59:27.099509 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.082018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38759 (local address 127.0.97.1:41559)
0926 02:59:27.082161 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.082166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.091364 (+ 9198us) server_negotiation.cc:407] Connection header received
0926 02:59:27.091603 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.091609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.091680 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.091743 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:27.092265 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.093264 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.096743 (+ 3479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.097994 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.098146 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.098294 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.098992 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.099180 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 02:59:27.100370 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.082177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41559 (local address 127.0.97.3:38759)
0926 02:59:27.091280 (+ 9103us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:27.091298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:27.091458 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:27.091913 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:27.091917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:27.091942 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:27.092106 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.092127 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.093431 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.093436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.095796 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.095816 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.098497 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.098505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.098774 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.098824 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 02:59:27.099719 (+ 895us) client_negotiation.cc:240] Negotiation successful
0926 02:59:27.099930 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9009,"threads_started":1}
I20260926 02:59:27.101297 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.075112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39625 (local address 127.0.97.1:42301)
0926 02:59:27.075992 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:27.076017 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:27.076168 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:27.076546 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:27.076550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:27.076603 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:27.077107 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.077124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.078515 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.078521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.080956 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:27.080970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.082872 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.082877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:27.083101 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.083155 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 02:59:27.100503 (+ 17348us) client_negotiation.cc:240] Negotiation successful
0926 02:59:27.100709 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":219,"thread_start_us":722,"threads_started":1}
I20260926 02:59:27.102640 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26707c4c1bcf4ed49aa1197455fd68a6" candidate_uuid: "66be2acee53b4ac9a40a9112844bcacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ad6e8240b240ff87e21c5ce911c007" is_pre_election: true
I20260926 02:59:27.103295 1240 raft_consensus.cc:2454] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66be2acee53b4ac9a40a9112844bcacd in term 0.
I20260926 02:59:27.103915 1444 leader_election.cc:288] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [CANDIDATE]: Term 1 election: Requested vote from peers 25ad6e8240b240ff87e21c5ce911c007 (127.0.97.2:39625), cecd023433c447f596909060be5254ef (127.0.97.3:37803)
I20260926 02:59:27.104261 1444 raft_consensus.cc:2781] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:27.104641 1444 raft_consensus.cc:686] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [term 1 LEADER]: Becoming Leader. State: Replica: 66be2acee53b4ac9a40a9112844bcacd, State: Running, Role: LEADER
I20260926 02:59:27.104949 1239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26707c4c1bcf4ed49aa1197455fd68a6" candidate_uuid: "66be2acee53b4ac9a40a9112844bcacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ad6e8240b240ff87e21c5ce911c007"
I20260926 02:59:27.105293 1239 raft_consensus.cc:3037] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:27.105178 1444 consensus_queue.cc:237] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } }
I20260926 02:59:27.110579 960 catalog_manager.cc:5495] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd reported cstate change: term changed from 0 to 1, leader changed from <none> to 66be2acee53b4ac9a40a9112844bcacd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "66be2acee53b4ac9a40a9112844bcacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ad6e8240b240ff87e21c5ce911c007" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66be2acee53b4ac9a40a9112844bcacd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cecd023433c447f596909060be5254ef" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37803 } health_report { overall_health: UNKNOWN } } }
W20260926 02:59:27.113400 1419 tablet.cc:2369] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:59:27.117960 1239 raft_consensus.cc:2454] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66be2acee53b4ac9a40a9112844bcacd in term 1.
I20260926 02:59:27.122715 1239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26707c4c1bcf4ed49aa1197455fd68a6" candidate_uuid: "cecd023433c447f596909060be5254ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ad6e8240b240ff87e21c5ce911c007" is_pre_election: true
I20260926 02:59:27.123090 1239 raft_consensus.cc:2379] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cecd023433c447f596909060be5254ef in current term 1: Already voted for candidate 66be2acee53b4ac9a40a9112844bcacd in this term.
I20260926 02:59:27.123903 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26707c4c1bcf4ed49aa1197455fd68a6" candidate_uuid: "cecd023433c447f596909060be5254ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66be2acee53b4ac9a40a9112844bcacd" is_pre_election: true
I20260926 02:59:27.124874 1306 leader_election.cc:302] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cecd023433c447f596909060be5254ef; no voters: 25ad6e8240b240ff87e21c5ce911c007, 66be2acee53b4ac9a40a9112844bcacd
I20260926 02:59:27.125344 1442 raft_consensus.cc:2726] T 26707c4c1bcf4ed49aa1197455fd68a6 P cecd023433c447f596909060be5254ef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:27.139700 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.125161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.0.97.1:41559)
0926 02:59:27.125332 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.125339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.126001 (+ 662us) server_negotiation.cc:407] Connection header received
0926 02:59:27.126101 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.126104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.126166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.126241 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:27.127853 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.128538 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.129784 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.130172 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.130263 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.134100 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:27.134110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:27.134123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:27.134150 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:27.137430 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.137980 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.137986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.137989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.138115 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.138434 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.138436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.138438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.138733 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:27.138949 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.139181 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.139352 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 02:59:27.148461 1286 tablet.cc:2369] T 26707c4c1bcf4ed49aa1197455fd68a6 P 25ad6e8240b240ff87e21c5ce911c007: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391567, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260926 02:59:27.177269 1152 tablet.cc:2369] T 26707c4c1bcf4ed49aa1197455fd68a6 P 66be2acee53b4ac9a40a9112844bcacd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:59:27.182862 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.168889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.0.97.1:41559)
0926 02:59:27.169137 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.169146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.169437 (+ 291us) server_negotiation.cc:407] Connection header received
0926 02:59:27.169643 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.169647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.169709 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.169822 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:27.171053 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.171762 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.173227 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.173667 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.173817 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.177382 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:27.177387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:27.177390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:27.177420 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:27.180358 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.181023 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.181031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.181034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.181151 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.181541 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.181546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.181549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.181818 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:27.182089 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.182351 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.182552 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:27.223687 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.211195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.0.97.1:41559)
0926 02:59:27.211426 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.211432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.211847 (+ 415us) server_negotiation.cc:407] Connection header received
0926 02:59:27.211979 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.211983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.212047 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.212130 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:27.213513 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.214196 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.215330 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.215733 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.215895 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.219452 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:27.219457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:27.219460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:27.219489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:27.221789 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.222316 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.222320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.222322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.222432 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.222691 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.222694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.222696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.222956 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:27.223128 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.223312 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.223454 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 02:59:27.225692 1066 tablet_service.cc:2908] Rejecting scan request for tablet 26707c4c1bcf4ed49aa1197455fd68a6: Uninitialized: clean time has not yet been initialized
I20260926 02:59:27.228752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1023
I20260926 02:59:27.239585 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.229566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.0.97.1:41559)
0926 02:59:27.229769 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.229777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.230019 (+ 242us) server_negotiation.cc:407] Connection header received
0926 02:59:27.230120 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.230122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.230172 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.230242 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:27.231235 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.231883 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.233048 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.233411 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.233505 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.235258 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:27.235262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:27.235265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:27.235316 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:27.237610 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.238150 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.238153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.238155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.238251 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.238532 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.238535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.238549 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.238768 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:27.238963 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.239117 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.239252 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:59:27.240150 1146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:27.331023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 1023
I20260926 02:59:27.348495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1157
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 02:59:27.362890 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.349276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.0.97.2:39625)
0926 02:59:27.349487 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.349492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.349509 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:59:27.349645 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.349649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.349708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.349817 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:27.350957 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.351658 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.352947 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.353493 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.353585 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.357451 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:27.357463 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:27.357479 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:27.357511 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:27.360231 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.360911 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.360918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.360921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.361096 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.361463 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.361468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.361472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.361802 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:27.362096 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.362362 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.362573 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 02:59:27.363513 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:27.452232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 1157
I20260926 02:59:27.468030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1290
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 02:59:27.481658 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:27.468744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.0.97.3:37803)
0926 02:59:27.468918 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:27.468934 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:27.468952 (+ 18us) server_negotiation.cc:407] Connection header received
0926 02:59:27.469139 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:27.469143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:27.469212 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:27.469292 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:27.470406 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.471110 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.472259 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:27.472631 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:27.472723 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:27.476601 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:27.476616 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:27.476633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:27.476664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:27.479270 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.479856 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.479864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.479866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.479978 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:27.480272 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:27.480275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:27.480277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:27.480617 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:27.480824 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:27.481156 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 02:59:27.481308 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 02:59:27.482271 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:27.568933 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 1290
I20260926 02:59:27.585079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 930
I20260926 02:59:27.586206 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:27.691190 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 930
2026-09-26T02:59:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4016 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[1468](info): set up 2 sockets
Sep 26 02:59:27 dist-test-slave-fccw krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260926 02:59:28.777096 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:28 dist-test-slave-fccw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:28.950347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37683
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:29.259455 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:29.259717 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:29.259886 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:29.266768 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:29.266853 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:59:29.266891 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:29.266942 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:29.267014 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:29.267125 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:29.276367 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37683
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:29.277608 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:29.278903 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:29.290478 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:29.290450 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:29.290532 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:29.291455 1481 server_base.cc:1034] running on GCE node
I20260926 02:59:29.292124 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:29.293399 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:29.294638 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790391569294580 us; error 44 us; skew 500 ppm
I20260926 02:59:29.294957 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:29 dist-test-slave-fccw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:29.297842 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:29.299078 1481 webserver.cc:460] Webserver started at http://127.0.97.62:42851/ using document root <none> and password file <none>
I20260926 02:59:29.299724 1481 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:29.299849 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:29.300159 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:29.302969 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "a597af5eeddd42a7b0febc2dd32fb907"
format_stamp: "Formatted at 2026-09-26 02:59:29 on dist-test-slave-fccw"
server_key: "12f8cba78ac612046fa6f5a5e1ba7659"
server_key_iv: "6c53acb29cc86622b4561702cdb9a60a"
server_key_version: "encryptionkey@0"
I20260926 02:59:29.303737 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "a597af5eeddd42a7b0febc2dd32fb907"
format_stamp: "Formatted at 2026-09-26 02:59:29 on dist-test-slave-fccw"
server_key: "12f8cba78ac612046fa6f5a5e1ba7659"
server_key_iv: "6c53acb29cc86622b4561702cdb9a60a"
server_key_version: "encryptionkey@0"
I20260926 02:59:29.308403 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 02:59:29.311728 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:29.312881 1481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 02:59:29.313079 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "a597af5eeddd42a7b0febc2dd32fb907"
format_stamp: "Formatted at 2026-09-26 02:59:29 on dist-test-slave-fccw"
server_key: "12f8cba78ac612046fa6f5a5e1ba7659"
server_key_iv: "6c53acb29cc86622b4561702cdb9a60a"
server_key_version: "encryptionkey@0"
I20260926 02:59:29.313309 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:29.325127 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:29.328552 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:29.328799 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:29.349427 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37683
I20260926 02:59:29.349447 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37683 every 8 connection(s)
I20260926 02:59:29.350754 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:29.354317 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:29.355997 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 1481
I20260926 02:59:29.356213 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:29.357028 388 external_mini_cluster.cc:1428] Setting key 38d2e18da0ec382e458cdf8fcb905c73
I20260926 02:59:29.366544 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907: Bootstrap starting.
I20260926 02:59:29.370354 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:29.371523 1558 log.cc:824] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907: Log is configured to *not* fsync() on all Append() calls
Sep 26 02:59:29 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:29.374228 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907: No bootstrap required, opened a new log
I20260926 02:59:29.379601 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:29.359113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.0.97.62:37683)
0926 02:59:29.359926 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:29.359938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:29.359976 (+ 38us) server_negotiation.cc:407] Connection header received
0926 02:59:29.361690 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:29.361711 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:29.362071 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:29.362375 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:29.364140 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.365563 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.367231 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.367863 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.367994 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:29.372573 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:29.372580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:29.372593 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:29.372629 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:29.375655 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:29.376361 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:29.376369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:29.376385 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:29.376542 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:29.376940 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:29.376944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:29.376946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:29.377493 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:29.377762 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:29.378222 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 02:59:29.378614 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":404,"threads_started":1}
I20260926 02:59:29.382193 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a597af5eeddd42a7b0febc2dd32fb907" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260926 02:59:29.382563 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:29.382685 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a597af5eeddd42a7b0febc2dd32fb907, State: Initialized, Role: FOLLOWER
I20260926 02:59:29.383188 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a597af5eeddd42a7b0febc2dd32fb907" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260926 02:59:29.383431 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:29.383570 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:29.383731 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:29.385973 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a597af5eeddd42a7b0febc2dd32fb907" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260926 02:59:29.386459 1558 leader_election.cc:302] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a597af5eeddd42a7b0febc2dd32fb907; no voters:
I20260926 02:59:29.387022 1558 leader_election.cc:288] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:29.387141 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:29.388445 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [term 1 LEADER]: Becoming Leader. State: Replica: a597af5eeddd42a7b0febc2dd32fb907, State: Running, Role: LEADER
I20260926 02:59:29.388978 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a597af5eeddd42a7b0febc2dd32fb907" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } }
I20260926 02:59:29.389343 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:29.392037 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a597af5eeddd42a7b0febc2dd32fb907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a597af5eeddd42a7b0febc2dd32fb907" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } } }
I20260926 02:59:29.392405 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:29.395825 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a597af5eeddd42a7b0febc2dd32fb907. Latest consensus state: current_term: 1 leader_uuid: "a597af5eeddd42a7b0febc2dd32fb907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a597af5eeddd42a7b0febc2dd32fb907" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37683 } } }
I20260926 02:59:29.396253 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:29.397236 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:29.401732 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:29.409896 1572 catalog_manager.cc:1347] Generated new cluster ID: b4f3ee24e3944a44bd6e18c01255add4
I20260926 02:59:29.410000 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:29.423278 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:29.424480 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:29.433375 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907: Generated new TSK 0
I20260926 02:59:29.434083 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:29.482497 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37683
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:29.796427 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:29.796710 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:29.796885 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:29.804260 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:29.804363 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:29.804447 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:59:29.804616 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:29.814086 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:29.815338 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:29.816800 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:29.829026 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:29.830302 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:29.830513 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:29.831513 1584 server_base.cc:1034] running on GCE node
I20260926 02:59:29.832198 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:29.833079 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:29.834311 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790391569834256 us; error 38 us; skew 500 ppm
I20260926 02:59:29.834616 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:29 dist-test-slave-fccw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:29.837774 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:29.839190 1584 webserver.cc:460] Webserver started at http://127.0.97.1:36505/ using document root <none> and password file <none>
I20260926 02:59:29.839897 1584 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:29.840021 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:29.840334 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:29.843189 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "ceafb7ea501f46cbadd3751d45b6d5d2"
format_stamp: "Formatted at 2026-09-26 02:59:29 on dist-test-slave-fccw"
server_key: "daf3aacecc38205a21629feae39b26ee"
server_key_iv: "4f35d9d19277da16facdf196636f4ae0"
server_key_version: "encryptionkey@0"
I20260926 02:59:29.843947 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ceafb7ea501f46cbadd3751d45b6d5d2"
format_stamp: "Formatted at 2026-09-26 02:59:29 on dist-test-slave-fccw"
server_key: "daf3aacecc38205a21629feae39b26ee"
server_key_iv: "4f35d9d19277da16facdf196636f4ae0"
server_key_version: "encryptionkey@0"
I20260926 02:59:29.849431 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 02:59:29.853045 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:29.854322 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:59:29.854491 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "ceafb7ea501f46cbadd3751d45b6d5d2"
format_stamp: "Formatted at 2026-09-26 02:59:29 on dist-test-slave-fccw"
server_key: "daf3aacecc38205a21629feae39b26ee"
server_key_iv: "4f35d9d19277da16facdf196636f4ae0"
server_key_version: "encryptionkey@0"
I20260926 02:59:29.854724 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:29.892024 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:29.896410 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:29.896685 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:29.899267 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:29.899389 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:29.899534 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:29.899648 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 02:59:29 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:29.922887 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:29.900035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35381 (local address 127.0.97.62:37683)
0926 02:59:29.900560 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:29.900564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:29.901903 (+ 1339us) server_negotiation.cc:407] Connection header received
0926 02:59:29.903955 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:29.903961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:29.904036 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:29.904155 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:29.906299 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.907553 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.910123 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.910735 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.910894 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:29.915877 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:29.915881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:29.915885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:29.915914 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:29.919027 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:29.919967 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:29.919974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:29.919980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:29.920120 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:29.920692 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:29.920699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:29.920704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:29.921176 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:29.921355 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:29.922023 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 02:59:29.922348 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":314,"threads_started":1}
I20260926 02:59:29.923414 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:29.900926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37683 (local address 127.0.97.1:35381)
0926 02:59:29.901750 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:29.901790 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:29.903705 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:29.904405 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:29.904419 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:29.905199 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:29.906016 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:29.906037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.907752 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.907759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:29.909991 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:29.910001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.911013 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.911020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:29.911246 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:29.911915 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:29.911940 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:29.915621 (+ 3681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:29.919160 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:29.919172 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:29.919200 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:29.919766 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:29.920234 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:29.920239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:29.920242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:29.920521 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:29.921315 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:29.921321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:29.921638 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 02:59:29.921860 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 02:59:29.922203 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":355,"threads_started":1}
I20260926 02:59:29.942164 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:29.929348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37683 (local address 127.0.97.1:36297)
0926 02:59:29.929877 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:29.929892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:29.930030 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:29.930336 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:29.930340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:29.930794 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:29.931053 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:29.931061 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.932065 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.932072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:29.933664 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:29.933678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.934385 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.934391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:29.934570 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:29.935272 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:29.935296 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:29.935912 (+ 616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:29.939052 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:29.939058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:29.939061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:29.939555 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:29.940030 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:29.940033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:29.940035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:29.940207 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:29.940888 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:29.940893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:29.941197 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 02:59:29.941379 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 02:59:29.941720 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":304,"threads_started":1}
I20260926 02:59:29.942286 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:29.929245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36297 (local address 127.0.97.62:37683)
0926 02:59:29.929407 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:29.929411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:29.929972 (+ 561us) server_negotiation.cc:407] Connection header received
0926 02:59:29.930154 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:29.930159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:29.930225 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:29.930319 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:29.931184 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.931951 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.933818 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:29.934214 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:29.934344 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:29.936070 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:29.936076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:29.936080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:29.936117 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:29.938933 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:29.939694 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:29.939702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:29.939708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:29.939906 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:29.940327 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:29.940334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:29.940337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:29.940761 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:29.940913 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:29.941742 (+ 829us) server_negotiation.cc:299] Negotiation successful
0926 02:59:29.941951 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:59:29.955571 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38801
I20260926 02:59:29.955593 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38801 every 8 connection(s)
I20260926 02:59:29.956830 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:29.959020 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37683
I20260926 02:59:29.959347 1720 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:29.959959 1720 heartbeater.cc:511] Master 127.0.97.62:37683 requested a full tablet report, sending...
I20260926 02:59:29.960886 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 1584
I20260926 02:59:29.961146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:29.961933 388 external_mini_cluster.cc:1428] Setting key f0d980e4e6120a700b48b5c0c9b10cc4
I20260926 02:59:29.962275 1513 ts_manager.cc:194] Registered new tserver with Master: ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1:38801)
I20260926 02:59:29.964326 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35381
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:29.999755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37683
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:30.300897 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:30.301168 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:30.301363 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:30.308189 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:30.308285 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:30.308446 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:59:30.308568 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:30.317974 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:30.319195 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:30.320439 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:30.333109 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:30.333155 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:30.333134 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:30.334448 1724 server_base.cc:1034] running on GCE node
I20260926 02:59:30.334982 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:30.335837 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:30.337080 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790391570337023 us; error 33 us; skew 500 ppm
I20260926 02:59:30.337366 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:30 dist-test-slave-fccw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:30.340168 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:30.341636 1724 webserver.cc:460] Webserver started at http://127.0.97.2:43187/ using document root <none> and password file <none>
I20260926 02:59:30.342463 1724 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:30.342571 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:30.343005 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:30.345793 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "fd25fd9fa0d140aa9a58a6a9254a129e"
format_stamp: "Formatted at 2026-09-26 02:59:30 on dist-test-slave-fccw"
server_key: "dac7d94eedd5937101c868004b38f04f"
server_key_iv: "3cb1fb949f3c1e3d4a431fc1fe80875b"
server_key_version: "encryptionkey@0"
I20260926 02:59:30.346481 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fd25fd9fa0d140aa9a58a6a9254a129e"
format_stamp: "Formatted at 2026-09-26 02:59:30 on dist-test-slave-fccw"
server_key: "dac7d94eedd5937101c868004b38f04f"
server_key_iv: "3cb1fb949f3c1e3d4a431fc1fe80875b"
server_key_version: "encryptionkey@0"
I20260926 02:59:30.351912 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:59:30.355962 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:30.357460 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:59:30.357617 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "fd25fd9fa0d140aa9a58a6a9254a129e"
format_stamp: "Formatted at 2026-09-26 02:59:30 on dist-test-slave-fccw"
server_key: "dac7d94eedd5937101c868004b38f04f"
server_key_iv: "3cb1fb949f3c1e3d4a431fc1fe80875b"
server_key_version: "encryptionkey@0"
I20260926 02:59:30.357841 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:30.382967 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:30.386641 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:30.386857 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:30.389348 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:30.389423 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:30.389626 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:30.389772 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:59:30 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:30.412566 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.389771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45287 (local address 127.0.97.62:37683)
0926 02:59:30.389974 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:30.389983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:30.391340 (+ 1357us) server_negotiation.cc:407] Connection header received
0926 02:59:30.393508 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:30.393516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:30.393602 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:30.393745 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:30.395284 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.396769 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.399376 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.399985 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.400155 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.404860 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:30.404869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:30.404875 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:30.404915 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:30.408623 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.409464 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.409470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.409473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.409624 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.410216 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.410224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.410227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.410622 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:30.410775 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:30.411759 (+ 984us) server_negotiation.cc:299] Negotiation successful
0926 02:59:30.412070 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:59:30.413306 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.390349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37683 (local address 127.0.97.2:45287)
0926 02:59:30.391164 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:30.391206 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:30.393256 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:30.394034 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:30.394045 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:30.394613 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:30.395110 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.395126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.396925 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.396928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.399187 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.399232 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.400295 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.400305 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.400721 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.401635 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:30.401674 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:30.404621 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:30.408789 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.408795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.408811 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.409319 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.409785 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.409792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.409797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.410082 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.410742 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:30.410749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:30.411265 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 02:59:30.411598 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 02:59:30.412067 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":402,"threads_started":1}
I20260926 02:59:30.434206 1810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.419501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37683 (local address 127.0.97.2:47897)
0926 02:59:30.420175 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:30.420195 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:30.420316 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:30.420683 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:30.420686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:30.421187 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:30.421668 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.421683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.422884 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.422887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.424447 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.424458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.425725 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.425737 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.425953 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.426611 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:30.426626 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:30.427266 (+ 640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:30.431324 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.431332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.431339 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.432038 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.432391 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.432396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.432400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.432521 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.433100 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:30.433105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:30.433349 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 02:59:30.433568 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 02:59:30.433906 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":383,"threads_started":1}
I20260926 02:59:30.434937 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.419501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47897 (local address 127.0.97.62:37683)
0926 02:59:30.419708 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:30.419716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:30.420281 (+ 565us) server_negotiation.cc:407] Connection header received
0926 02:59:30.420430 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:30.420438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:30.420540 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:30.420664 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:30.421865 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.422728 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.424605 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.425481 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.425616 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.427451 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:30.427460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:30.427467 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:30.427515 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:30.431182 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.432160 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.432164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.432166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.432278 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.432626 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.432634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.432637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.432931 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:30.433130 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:30.434505 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0926 02:59:30.434661 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 02:59:30.445580 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36169
I20260926 02:59:30.445601 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36169 every 8 connection(s)
I20260926 02:59:30.446777 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:30.448818 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37683
I20260926 02:59:30.449070 1859 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:30.449674 1859 heartbeater.cc:511] Master 127.0.97.62:37683 requested a full tablet report, sending...
I20260926 02:59:30.450894 1513 ts_manager.cc:194] Registered new tserver with Master: fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2:36169)
I20260926 02:59:30.451622 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45287
I20260926 02:59:30.455291 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 1724
I20260926 02:59:30.455441 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:30.455871 388 external_mini_cluster.cc:1428] Setting key f0edf364c7ffb95b2be2422a6112da65
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:30.492365 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37683
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:30.798592 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:30.798889 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:30.799058 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:30.806208 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:30.806370 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:30.806527 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:59:30.806679 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:30.816025 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:30.817381 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:30.818660 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:30.831218 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:30.831135 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:30.831178 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:30.831959 1863 server_base.cc:1034] running on GCE node
I20260926 02:59:30.832450 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:30.833319 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:30.834547 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790391570834468 us; error 58 us; skew 500 ppm
I20260926 02:59:30.834820 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:30 dist-test-slave-fccw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:30.837839 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:30.839080 1863 webserver.cc:460] Webserver started at http://127.0.97.3:38445/ using document root <none> and password file <none>
I20260926 02:59:30.839774 1863 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:30.839900 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:30.840209 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:30.843073 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "db54b07b493c45bd911b6138868a7513"
format_stamp: "Formatted at 2026-09-26 02:59:30 on dist-test-slave-fccw"
server_key: "6ff04319a52298e7e74c37d1fb1a063a"
server_key_iv: "e2d7f81dc97a33c3421a49c87472c6c1"
server_key_version: "encryptionkey@0"
I20260926 02:59:30.843842 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "db54b07b493c45bd911b6138868a7513"
format_stamp: "Formatted at 2026-09-26 02:59:30 on dist-test-slave-fccw"
server_key: "6ff04319a52298e7e74c37d1fb1a063a"
server_key_iv: "e2d7f81dc97a33c3421a49c87472c6c1"
server_key_version: "encryptionkey@0"
I20260926 02:59:30.848692 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 02:59:30.852691 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:30.854368 1863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 02:59:30.854522 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "db54b07b493c45bd911b6138868a7513"
format_stamp: "Formatted at 2026-09-26 02:59:30 on dist-test-slave-fccw"
server_key: "6ff04319a52298e7e74c37d1fb1a063a"
server_key_iv: "e2d7f81dc97a33c3421a49c87472c6c1"
server_key_version: "encryptionkey@0"
I20260926 02:59:30.854734 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:30.873925 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:30.877669 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:30.877944 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:30.880347 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:30.880450 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:30.880627 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:30.880704 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:59:30 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:30.903582 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.881057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34203 (local address 127.0.97.62:37683)
0926 02:59:30.881238 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:30.881254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:30.882624 (+ 1370us) server_negotiation.cc:407] Connection header received
0926 02:59:30.885502 (+ 2878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:30.885510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:30.885595 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:30.885735 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:30.887747 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.888845 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.891423 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.892106 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.892239 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.896586 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:30.896599 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:30.896602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:30.896631 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:30.899746 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.900695 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.900699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.900701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.900813 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.901412 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.901418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.901421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.901710 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:30.901884 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:30.902988 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0926 02:59:30.903217 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:59:30.904873 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.881571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37683 (local address 127.0.97.3:34203)
0926 02:59:30.882435 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:30.882488 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:30.885151 (+ 2663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:30.885983 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:30.885997 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:30.886736 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:30.887572 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.887590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.889055 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.889060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.891275 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.891285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.892355 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.892363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.892573 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.893367 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:30.893407 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:30.896371 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:30.899872 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.899881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.899900 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.900490 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.900921 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.900927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.900931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.901254 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.901967 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:30.901974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:30.902488 (+ 514us) client_negotiation.cc:769] Sending connection context
0926 02:59:30.902870 (+ 382us) client_negotiation.cc:240] Negotiation successful
0926 02:59:30.903443 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
I20260926 02:59:30.925776 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.911331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34859 (local address 127.0.97.62:37683)
0926 02:59:30.911807 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:30.911815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:30.912309 (+ 494us) server_negotiation.cc:407] Connection header received
0926 02:59:30.912529 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:30.912534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:30.912599 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:30.912707 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:30.914088 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.914742 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.916022 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.916577 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.916702 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.919230 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:30.919236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:30.919241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:30.919278 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:30.922833 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.923798 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.923805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.923811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.923938 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.924373 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.924376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.924378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.924665 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:30.924795 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:30.925438 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 02:59:30.925573 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338}
I20260926 02:59:30.926092 1966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.911377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37683 (local address 127.0.97.3:34859)
0926 02:59:30.912187 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:30.912211 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:30.912366 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:30.912728 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:30.912733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:30.913367 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:30.913952 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.913969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.914869 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.914873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.915869 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:30.915897 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.916793 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.916802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:30.917103 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.917872 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:30.917894 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:30.919008 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:30.922981 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.922988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.922991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.923672 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.924051 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:30.924058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:30.924064 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:30.924258 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:30.924811 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:30.924819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:30.925075 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 02:59:30.925337 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 02:59:30.925616 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":483,"threads_started":1}
I20260926 02:59:30.930297 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33485
I20260926 02:59:30.930327 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33485 every 8 connection(s)
I20260926 02:59:30.931443 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:30.933259 1998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37683
I20260926 02:59:30.933462 1998 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:30.934041 1998 heartbeater.cc:511] Master 127.0.97.62:37683 requested a full tablet report, sending...
I20260926 02:59:30.935242 1513 ts_manager.cc:194] Registered new tserver with Master: db54b07b493c45bd911b6138868a7513 (127.0.97.3:33485)
I20260926 02:59:30.935956 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34203
I20260926 02:59:30.937201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 1863
I20260926 02:59:30.937381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:30.937896 388 external_mini_cluster.cc:1428] Setting key 45da69338f08b2cdcd661dfbd1302c10
I20260926 02:59:30.940435 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:59:30 dist-test-slave-fccw krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260926 02:59:30.966934 1720 heartbeater.cc:503] Master 127.0.97.62:37683 was elected leader, sending a full tablet report...
Sep 26 02:59:30 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:30.982748 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:30.969553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.0.97.62:37683)
0926 02:59:30.969763 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:30.969770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:30.970035 (+ 265us) server_negotiation.cc:407] Connection header received
0926 02:59:30.970236 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:30.970251 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:30.970316 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:30.970416 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:30.971782 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.972485 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.973765 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:30.974167 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:30.974313 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:30.977963 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:30.977968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:30.977971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:30.978000 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:30.980194 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.980813 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.980820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.980824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.980935 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:30.981392 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:30.981396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:30.981400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:30.981647 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:30.981884 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:30.982181 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 02:59:30.982408 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:59:30.986340 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:59:30.993140 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:31.013669 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.004108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33485 (local address 127.0.0.1:39470)
0926 02:59:31.004865 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.004938 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.005139 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.006121 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.006127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.006158 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.006595 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.006607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.008545 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.008556 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.010838 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.010856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.012651 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.012658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.012842 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.012888 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.013049 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.013220 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":30,"thread_start_us":354,"threads_started":1}
I20260926 02:59:31.014093 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.004328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.0.97.3:33485)
0926 02:59:31.005087 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.005099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.005132 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:59:31.005345 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.005353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.005624 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.005816 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.006834 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.008421 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.010997 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.012458 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.012593 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.012856 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.013275 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.013534 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":361,"threads_started":1}
I20260926 02:59:31.014117 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.005328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38801 (local address 127.0.0.1:58506)
0926 02:59:31.005854 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.005873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.006000 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.007974 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.007978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.007999 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.008405 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.008417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.010001 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.010007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.011835 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.011845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.013384 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.013390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.013545 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.013575 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.013648 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.013788 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":46,"thread_start_us":248,"threads_started":1}
I20260926 02:59:31.014964 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.005597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.0.97.1:38801)
0926 02:59:31.007342 (+ 1745us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.007353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.007382 (+ 29us) server_negotiation.cc:407] Connection header received
0926 02:59:31.007527 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.007533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.007806 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.007989 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.008613 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.009860 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.011972 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.013163 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.014312 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.014494 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.014553 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.014698 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1594,"thread_start_us":1480,"threads_started":1}
I20260926 02:59:31.016211 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.005882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36169 (local address 127.0.0.1:35082)
0926 02:59:31.006692 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.006710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.006870 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.008228 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.008238 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.008263 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.008655 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.008683 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.010433 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.010438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.012464 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.012475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.015431 (+ 2956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.015435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.015547 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.015566 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.015756 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.015848 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":2,"spinlock_wait_cycles":66944,"thread_start_us":616,"threads_started":1}
I20260926 02:59:31.016359 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.006327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.0.97.2:36169)
0926 02:59:31.007695 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.007703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.007726 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:59:31.007814 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.007819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.008027 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.008225 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.008860 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.010291 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.013880 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.015264 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.015397 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.015562 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.015818 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.016018 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":1112,"threads_started":1}
I20260926 02:59:31.018046 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 93f4fb14c5da413880304bc2657e253c (DEFAULT_TABLE table=test-table [id=df6c6fcbfaf34b12a867c454a2729d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:31.019246 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93f4fb14c5da413880304bc2657e253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.019188 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 93f4fb14c5da413880304bc2657e253c (DEFAULT_TABLE table=test-table [id=df6c6fcbfaf34b12a867c454a2729d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:31.020345 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93f4fb14c5da413880304bc2657e253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.024060 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 93f4fb14c5da413880304bc2657e253c (DEFAULT_TABLE table=test-table [id=df6c6fcbfaf34b12a867c454a2729d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:31.025521 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93f4fb14c5da413880304bc2657e253c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.031113 2017 tablet_bootstrap.cc:492] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Bootstrap starting.
I20260926 02:59:31.032950 2018 tablet_bootstrap.cc:492] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Bootstrap starting.
I20260926 02:59:31.034639 2017 tablet_bootstrap.cc:654] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.036116 2018 tablet_bootstrap.cc:654] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.036315 2017 log.cc:824] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:31.037307 2018 log.cc:824] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:31.039074 2017 tablet_bootstrap.cc:492] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: No bootstrap required, opened a new log
I20260926 02:59:31.039121 2020 tablet_bootstrap.cc:492] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Bootstrap starting.
I20260926 02:59:31.039363 2017 ts_tablet_manager.cc:1397] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 02:59:31.040192 2018 tablet_bootstrap.cc:492] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: No bootstrap required, opened a new log
I20260926 02:59:31.040431 2018 ts_tablet_manager.cc:1397] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 02:59:31.041960 2020 tablet_bootstrap.cc:654] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.042851 2020 log.cc:824] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:31.046945 2020 tablet_bootstrap.cc:492] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: No bootstrap required, opened a new log
I20260926 02:59:31.047188 2020 ts_tablet_manager.cc:1397] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 02:59:31.050370 2017 raft_consensus.cc:348] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.050801 2017 raft_consensus.cc:374] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.050945 2017 raft_consensus.cc:729] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd25fd9fa0d140aa9a58a6a9254a129e, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.050987 2018 raft_consensus.cc:348] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.051519 2018 raft_consensus.cc:374] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.051651 2018 raft_consensus.cc:729] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db54b07b493c45bd911b6138868a7513, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.051656 2017 consensus_queue.cc:260] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.052337 2018 consensus_queue.cc:260] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.052675 1859 heartbeater.cc:503] Master 127.0.97.62:37683 was elected leader, sending a full tablet report...
I20260926 02:59:31.053499 2017 ts_tablet_manager.cc:1428] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260926 02:59:31.055008 2018 ts_tablet_manager.cc:1428] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 02:59:31.055630 2020 raft_consensus.cc:348] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.055938 2020 raft_consensus.cc:374] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.056026 2020 raft_consensus.cc:729] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceafb7ea501f46cbadd3751d45b6d5d2, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.056471 2020 consensus_queue.cc:260] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.056828 1998 heartbeater.cc:503] Master 127.0.97.62:37683 was elected leader, sending a full tablet report...
I20260926 02:59:31.060010 2020 ts_tablet_manager.cc:1428] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 02:59:31.130762 2024 raft_consensus.cc:482] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:31.131007 2024 raft_consensus.cc:504] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.132510 2024 leader_election.cc:288] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1:38801), fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2:36169)
I20260926 02:59:31.138178 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.132916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38801 (local address 127.0.97.3:58173)
0926 02:59:31.133113 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.133129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.133230 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.133774 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.133783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.133808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.133988 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.133996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.135010 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.135018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.136282 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.136294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.137407 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.137413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.137638 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.137661 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.137785 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.137911 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":49}
I20260926 02:59:31.138154 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.132860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58173 (local address 127.0.97.1:38801)
0926 02:59:31.133087 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.133096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.133198 (+ 102us) server_negotiation.cc:407] Connection header received
0926 02:59:31.133359 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.133365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.133455 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.133553 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.134118 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.134890 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.136434 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.137245 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.137390 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.137481 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.137808 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.137919 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 02:59:31.139369 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.133359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36169 (local address 127.0.97.3:44345)
0926 02:59:31.133975 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.133988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.134118 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.134433 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.134437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.134459 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.135310 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.135318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.136273 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.136281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.137593 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.137607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.138856 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.138860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.138965 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.138986 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.139065 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.139134 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":20,"thread_start_us":343,"threads_started":1}
I20260926 02:59:31.139182 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4fb14c5da413880304bc2657e253c" candidate_uuid: "db54b07b493c45bd911b6138868a7513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" is_pre_election: true
I20260926 02:59:31.139490 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.133482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44345 (local address 127.0.97.2:36169)
0926 02:59:31.133677 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.133684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.134076 (+ 392us) server_negotiation.cc:407] Connection header received
0926 02:59:31.134261 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.134266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.134330 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.134421 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.135459 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.136142 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.137741 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.138674 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.138822 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.138920 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.139117 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.139253 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:59:31.139719 1675 raft_consensus.cc:2454] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db54b07b493c45bd911b6138868a7513 in term 0.
I20260926 02:59:31.140278 1880 leader_election.cc:302] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceafb7ea501f46cbadd3751d45b6d5d2, db54b07b493c45bd911b6138868a7513; no voters:
I20260926 02:59:31.140309 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4fb14c5da413880304bc2657e253c" candidate_uuid: "db54b07b493c45bd911b6138868a7513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" is_pre_election: true
I20260926 02:59:31.140648 2024 raft_consensus.cc:2781] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:31.140790 2024 raft_consensus.cc:482] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:31.140901 2024 raft_consensus.cc:3037] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.140825 1814 raft_consensus.cc:2454] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db54b07b493c45bd911b6138868a7513 in term 0.
I20260926 02:59:31.143216 2024 raft_consensus.cc:504] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.143766 2024 leader_election.cc:288] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [CANDIDATE]: Term 1 election: Requested vote from peers ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1:38801), fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2:36169)
I20260926 02:59:31.144114 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4fb14c5da413880304bc2657e253c" candidate_uuid: "db54b07b493c45bd911b6138868a7513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2"
I20260926 02:59:31.144182 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93f4fb14c5da413880304bc2657e253c" candidate_uuid: "db54b07b493c45bd911b6138868a7513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e"
I20260926 02:59:31.144295 1675 raft_consensus.cc:3037] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.144390 1814 raft_consensus.cc:3037] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.146466 1675 raft_consensus.cc:2454] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db54b07b493c45bd911b6138868a7513 in term 1.
I20260926 02:59:31.146829 1880 leader_election.cc:302] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceafb7ea501f46cbadd3751d45b6d5d2, db54b07b493c45bd911b6138868a7513; no voters:
I20260926 02:59:31.147102 2024 raft_consensus.cc:2781] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:31.147123 1814 raft_consensus.cc:2454] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db54b07b493c45bd911b6138868a7513 in term 1.
I20260926 02:59:31.147604 2024 raft_consensus.cc:686] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 1 LEADER]: Becoming Leader. State: Replica: db54b07b493c45bd911b6138868a7513, State: Running, Role: LEADER
I20260926 02:59:31.148052 2024 consensus_queue.cc:237] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } }
I20260926 02:59:31.152137 1512 catalog_manager.cc:5495] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 reported cstate change: term changed from 0 to 1, leader changed from <none> to db54b07b493c45bd911b6138868a7513 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "db54b07b493c45bd911b6138868a7513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } health_report { overall_health: UNKNOWN } } }
I20260926 02:59:31.172346 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.167477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.0.97.3:33485)
0926 02:59:31.167711 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.167723 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.167889 (+ 166us) server_negotiation.cc:407] Connection header received
0926 02:59:31.168110 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.168114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.168186 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.168296 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:31.168588 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.169334 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.170553 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.170925 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.171010 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.171437 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.171906 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.172012 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.172106 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.172161 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:59:31.179888 1814 raft_consensus.cc:1261] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Refusing update from remote peer db54b07b493c45bd911b6138868a7513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:31.179924 1675 raft_consensus.cc:1261] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Refusing update from remote peer db54b07b493c45bd911b6138868a7513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:31.180788 2027 consensus_queue.cc:1035] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:31.181190 2024 consensus_queue.cc:1035] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:59:31.183188 1999 tablet.cc:2369] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:59:31.192054 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333443875543834624 to 7333443875425845248. Current Snapshot: MvccSnapshot[applied={T|T < 7333443875543834624}]
I20260926 02:59:31.194666 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333443875543834624 to 7333443875425845248. Current Snapshot: MvccSnapshot[applied={T|T < 7333443875543834624}]
I20260926 02:59:31.200731 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333443875543834624 to 7333443875425845248. Current Snapshot: MvccSnapshot[applied={T|T < 7333443875543834624 or (T in {7333443875543834624})}]
I20260926 02:59:31.211334 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:31.211676 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:31.214818 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907: Sending DeleteTablet for 3 replicas of tablet 93f4fb14c5da413880304bc2657e253c
I20260926 02:59:31.215796 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 93f4fb14c5da413880304bc2657e253c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39470
I20260926 02:59:31.216089 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 93f4fb14c5da413880304bc2657e253c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35082
I20260926 02:59:31.216536 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 93f4fb14c5da413880304bc2657e253c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58506
I20260926 02:59:31.218724 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:59:31.219527 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:31.225608 2041 tablet_replica.cc:330] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: stopping tablet replica
I20260926 02:59:31.226255 2041 raft_consensus.cc:2229] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:31.226925 2041 raft_consensus.cc:2258] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:31.227963 2039 tablet_replica.cc:330] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: stopping tablet replica
I20260926 02:59:31.228300 2039 raft_consensus.cc:2229] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:31.228897 2039 raft_consensus.cc:2258] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:31.229328 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 80148913ff95455996a63ab2ba112399 (DEFAULT_TABLE table=test-table [id=cefb0a6e33b94117a5571891c693f995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:31.229499 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 80148913ff95455996a63ab2ba112399 (DEFAULT_TABLE table=test-table [id=cefb0a6e33b94117a5571891c693f995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:31.229772 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80148913ff95455996a63ab2ba112399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.230055 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80148913ff95455996a63ab2ba112399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.230479 2041 ts_tablet_manager.cc:1905] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:31.231362 2040 tablet_replica.cc:330] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: stopping tablet replica
I20260926 02:59:31.231750 2040 raft_consensus.cc:2229] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:31.233151 2040 raft_consensus.cc:2258] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:31.235795 2040 ts_tablet_manager.cc:1905] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:31.236611 2020 tablet_bootstrap.cc:492] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Bootstrap starting.
I20260926 02:59:31.237504 2017 tablet_bootstrap.cc:492] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Bootstrap starting.
I20260926 02:59:31.238198 2020 tablet_bootstrap.cc:654] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.238724 2017 tablet_bootstrap.cc:654] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.240271 2020 tablet_bootstrap.cc:492] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: No bootstrap required, opened a new log
I20260926 02:59:31.240474 2020 ts_tablet_manager.cc:1397] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:59:31.241371 2020 raft_consensus.cc:348] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.241619 2020 raft_consensus.cc:374] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.241755 2020 raft_consensus.cc:729] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceafb7ea501f46cbadd3751d45b6d5d2, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.242074 2020 consensus_queue.cc:260] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.242758 2020 ts_tablet_manager.cc:1428] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:31.244925 2041 ts_tablet_manager.cc:1918] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:31.245151 2041 log.cc:1196] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/93f4fb14c5da413880304bc2657e253c
I20260926 02:59:31.245394 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 80148913ff95455996a63ab2ba112399 (DEFAULT_TABLE table=test-table [id=cefb0a6e33b94117a5571891c693f995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:31.245761 2041 ts_tablet_manager.cc:1939] T 93f4fb14c5da413880304bc2657e253c P ceafb7ea501f46cbadd3751d45b6d5d2: Deleting consensus metadata
I20260926 02:59:31.245993 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80148913ff95455996a63ab2ba112399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.247548 1498 catalog_manager.cc:4841] TS ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1:38801): tablet 93f4fb14c5da413880304bc2657e253c (table test-table [id=df6c6fcbfaf34b12a867c454a2729d89]) successfully deleted
I20260926 02:59:31.248749 2040 ts_tablet_manager.cc:1918] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:31.248909 2040 log.cc:1196] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/93f4fb14c5da413880304bc2657e253c
I20260926 02:59:31.249374 2040 ts_tablet_manager.cc:1939] T 93f4fb14c5da413880304bc2657e253c P fd25fd9fa0d140aa9a58a6a9254a129e: Deleting consensus metadata
I20260926 02:59:31.250725 1498 catalog_manager.cc:4841] TS fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2:36169): tablet 93f4fb14c5da413880304bc2657e253c (table test-table [id=df6c6fcbfaf34b12a867c454a2729d89]) successfully deleted
I20260926 02:59:31.245621 2039 ts_tablet_manager.cc:1905] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:31.252243 2018 tablet_bootstrap.cc:492] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: Bootstrap starting.
I20260926 02:59:31.255415 2018 tablet_bootstrap.cc:654] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.256453 2025 raft_consensus.cc:482] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:31.256688 2025 raft_consensus.cc:504] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.258653 2025 leader_election.cc:288] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2:36169), db54b07b493c45bd911b6138868a7513 (127.0.97.3:33485)
I20260926 02:59:31.258828 2017 tablet_bootstrap.cc:492] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: No bootstrap required, opened a new log
I20260926 02:59:31.258968 2017 ts_tablet_manager.cc:1397] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.012s
I20260926 02:59:31.259863 2017 raft_consensus.cc:348] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.260143 2018 tablet_bootstrap.cc:492] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: No bootstrap required, opened a new log
I20260926 02:59:31.260143 2017 raft_consensus.cc:374] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.260241 2017 raft_consensus.cc:729] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd25fd9fa0d140aa9a58a6a9254a129e, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.260309 2018 ts_tablet_manager.cc:1397] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:59:31.260588 2017 consensus_queue.cc:260] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.261266 2017 ts_tablet_manager.cc:1428] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:31.261200 2018 raft_consensus.cc:348] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.261493 2018 raft_consensus.cc:374] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.261698 2018 raft_consensus.cc:729] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db54b07b493c45bd911b6138868a7513, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.261999 2018 consensus_queue.cc:260] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.262603 2018 ts_tablet_manager.cc:1428] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 02:59:31.263468 2039 ts_tablet_manager.cc:1918] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:31.263702 2039 log.cc:1196] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/93f4fb14c5da413880304bc2657e253c
I20260926 02:59:31.264271 2039 ts_tablet_manager.cc:1939] T 93f4fb14c5da413880304bc2657e253c P db54b07b493c45bd911b6138868a7513: Deleting consensus metadata
I20260926 02:59:31.265839 1498 catalog_manager.cc:4841] TS db54b07b493c45bd911b6138868a7513 (127.0.97.3:33485): tablet 93f4fb14c5da413880304bc2657e253c (table test-table [id=df6c6fcbfaf34b12a867c454a2729d89]) successfully deleted
I20260926 02:59:31.277292 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.259419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54705 (local address 127.0.97.2:36169)
0926 02:59:31.259562 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.259567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.269563 (+ 9996us) server_negotiation.cc:407] Connection header received
0926 02:59:31.269846 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.269851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.269931 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.269995 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.270769 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.271858 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.274451 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.276154 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.276312 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.276444 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.276801 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.276930 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 02:59:31.277801 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.258980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36169 (local address 127.0.97.1:54705)
0926 02:59:31.269459 (+ 10479us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.269480 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.269684 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.270199 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.270203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.270229 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.270582 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.270593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.272052 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.272057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.274264 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.274278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.276389 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.276395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.276596 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.276632 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.277288 (+ 656us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.277429 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10334,"thread_start_us":10197,"threads_started":1}
I20260926 02:59:31.278885 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80148913ff95455996a63ab2ba112399" candidate_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" is_pre_election: true
I20260926 02:59:31.279138 1814 raft_consensus.cc:2454] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceafb7ea501f46cbadd3751d45b6d5d2 in term 0.
I20260926 02:59:31.279659 1601 leader_election.cc:302] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceafb7ea501f46cbadd3751d45b6d5d2, fd25fd9fa0d140aa9a58a6a9254a129e; no voters:
I20260926 02:59:31.280063 2025 raft_consensus.cc:2781] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:31.280224 2025 raft_consensus.cc:482] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:31.280352 2025 raft_consensus.cc:3037] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.283016 2025 raft_consensus.cc:504] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.283684 2025 leader_election.cc:288] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [CANDIDATE]: Term 1 election: Requested vote from peers fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2:36169), db54b07b493c45bd911b6138868a7513 (127.0.97.3:33485)
I20260926 02:59:31.284211 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80148913ff95455996a63ab2ba112399" candidate_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e"
I20260926 02:59:31.284410 1814 raft_consensus.cc:3037] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.284466 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.269707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43847 (local address 127.0.97.3:33485)
0926 02:59:31.269894 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.269901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.277052 (+ 7151us) server_negotiation.cc:407] Connection header received
0926 02:59:31.277292 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.277298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.277377 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.277452 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.278196 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.279452 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.282126 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.283423 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.283547 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.283648 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.284057 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.284184 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 02:59:31.284498 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.269645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33485 (local address 127.0.97.1:43847)
0926 02:59:31.276935 (+ 7290us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.276951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.277121 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.277616 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.277621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.277644 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.277993 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.278006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.279571 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.279577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.281945 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.281962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.283643 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.283649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.283855 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.283889 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.284015 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.284165 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7184,"thread_start_us":6989,"threads_started":1}
I20260926 02:59:31.285739 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80148913ff95455996a63ab2ba112399" candidate_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db54b07b493c45bd911b6138868a7513" is_pre_election: true
I20260926 02:59:31.285894 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80148913ff95455996a63ab2ba112399" candidate_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db54b07b493c45bd911b6138868a7513"
I20260926 02:59:31.286214 1948 raft_consensus.cc:2454] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceafb7ea501f46cbadd3751d45b6d5d2 in term 0.
I20260926 02:59:31.287278 1814 raft_consensus.cc:2454] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceafb7ea501f46cbadd3751d45b6d5d2 in term 1.
I20260926 02:59:31.287632 1601 leader_election.cc:302] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ceafb7ea501f46cbadd3751d45b6d5d2, fd25fd9fa0d140aa9a58a6a9254a129e; no voters: db54b07b493c45bd911b6138868a7513
I20260926 02:59:31.287904 2025 raft_consensus.cc:2781] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:31.288335 2025 raft_consensus.cc:686] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 LEADER]: Becoming Leader. State: Replica: ceafb7ea501f46cbadd3751d45b6d5d2, State: Running, Role: LEADER
I20260926 02:59:31.288735 2025 consensus_queue.cc:237] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.292181 1513 catalog_manager.cc:5495] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } health_report { overall_health: UNKNOWN } } }
Sep 26 02:59:31 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:31.346952 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.331300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56364 (local address 127.0.97.62:37683)
0926 02:59:31.331495 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.331503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.332080 (+ 577us) server_negotiation.cc:407] Connection header received
0926 02:59:31.332266 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.332270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.332325 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.332405 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:31.334348 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.335161 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.336914 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.337438 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.337523 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.341351 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:31.341359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:31.341368 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:31.341424 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:31.344389 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.345053 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.345059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.345062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.345195 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.345567 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.345572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.345575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.345812 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:31.345955 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.346345 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.346646 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":59}
I20260926 02:59:31.346984 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.331314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37683 (local address 127.0.0.1:56364)
0926 02:59:31.331951 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.331977 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.332155 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.332462 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.332466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.333851 (+ 1385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:31.334193 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.334206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.335279 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.335284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.336791 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.336804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.337601 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.337607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.337779 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.338428 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:31.338450 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:31.341185 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:31.344527 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:31.344538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:31.344545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:31.344917 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:31.345313 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:31.345317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:31.345321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:31.345460 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:31.345932 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:31.345939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:31.346096 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.346261 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.346526 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":348,"threads_started":1}
I20260926 02:59:31.352079 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56364:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:59:31.353215 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:31.360764 1649 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd146feb1a41ccad021f7491f93c1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3aa63b9b5be4c71985bcbe59d192474]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:59:31.360715 1928 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd146feb1a41ccad021f7491f93c1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3aa63b9b5be4c71985bcbe59d192474]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:59:31.361263 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5dd146feb1a41ccad021f7491f93c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.361756 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5dd146feb1a41ccad021f7491f93c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.365602 1789 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd146feb1a41ccad021f7491f93c1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3aa63b9b5be4c71985bcbe59d192474]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:59:31.366137 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5dd146feb1a41ccad021f7491f93c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:31.366887 2020 tablet_bootstrap.cc:492] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: Bootstrap starting.
I20260926 02:59:31.368301 2020 tablet_bootstrap.cc:654] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.370155 2020 tablet_bootstrap.cc:492] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: No bootstrap required, opened a new log
I20260926 02:59:31.370355 2020 ts_tablet_manager.cc:1397] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260926 02:59:31.370960 2020 raft_consensus.cc:348] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.371266 2020 raft_consensus.cc:374] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.371685 2020 raft_consensus.cc:729] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceafb7ea501f46cbadd3751d45b6d5d2, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.372004 2020 consensus_queue.cc:260] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.372572 2020 ts_tablet_manager.cc:1428] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 02:59:31.372435 2025 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.372794 2025 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:31.377064 2018 tablet_bootstrap.cc:492] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: Bootstrap starting.
I20260926 02:59:31.378494 2018 tablet_bootstrap.cc:654] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.378852 2017 tablet_bootstrap.cc:492] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: Bootstrap starting.
I20260926 02:59:31.380460 2017 tablet_bootstrap.cc:654] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:31.380712 2018 tablet_bootstrap.cc:492] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: No bootstrap required, opened a new log
I20260926 02:59:31.381107 2018 ts_tablet_manager.cc:1397] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260926 02:59:31.382164 2018 raft_consensus.cc:348] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.382462 2018 raft_consensus.cc:374] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.382580 2018 raft_consensus.cc:729] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db54b07b493c45bd911b6138868a7513, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.383513 2018 consensus_queue.cc:260] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.383975 2024 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.384241 2024 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:31.386653 2018 ts_tablet_manager.cc:1428] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260926 02:59:31.394537 2017 tablet_bootstrap.cc:492] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: No bootstrap required, opened a new log
I20260926 02:59:31.394719 2017 ts_tablet_manager.cc:1397] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260926 02:59:31.395443 2017 raft_consensus.cc:348] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.395690 2017 raft_consensus.cc:374] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:31.395766 2017 raft_consensus.cc:729] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd25fd9fa0d140aa9a58a6a9254a129e, State: Initialized, Role: FOLLOWER
I20260926 02:59:31.396402 2017 consensus_queue.cc:260] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.397019 2017 ts_tablet_manager.cc:1428] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 02:59:31.396912 2023 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.397348 2023 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:31.405714 2023 raft_consensus.cc:482] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:31.405889 2023 raft_consensus.cc:504] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.407459 2023 leader_election.cc:288] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1:38801), db54b07b493c45bd911b6138868a7513 (127.0.97.3:33485)
I20260926 02:59:31.417208 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.407802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38335 (local address 127.0.97.1:38801)
0926 02:59:31.407989 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.407996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.408582 (+ 586us) server_negotiation.cc:407] Connection header received
0926 02:59:31.408908 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.408917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.409060 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.409138 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.410142 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.411582 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.414695 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.416064 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.416184 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.416282 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.416759 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.416889 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 02:59:31.417366 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.407784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38801 (local address 127.0.97.2:38335)
0926 02:59:31.408493 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.408511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.408694 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.409393 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.409400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.409433 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.409920 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.409934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.411834 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.411842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.414537 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.414550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.416296 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.416302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.416576 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.416611 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.416713 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.416835 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":17,"thread_start_us":402,"threads_started":1}
I20260926 02:59:31.418133 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd146feb1a41ccad021f7491f93c1b" candidate_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" is_pre_election: true
I20260926 02:59:31.418397 1675 raft_consensus.cc:2454] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd25fd9fa0d140aa9a58a6a9254a129e in term 0.
I20260926 02:59:31.418890 1741 leader_election.cc:302] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceafb7ea501f46cbadd3751d45b6d5d2, fd25fd9fa0d140aa9a58a6a9254a129e; no voters:
I20260926 02:59:31.419335 2023 raft_consensus.cc:2781] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:31.419440 2023 raft_consensus.cc:482] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:31.419548 2023 raft_consensus.cc:3037] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.422444 2023 raft_consensus.cc:504] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.422963 2023 leader_election.cc:288] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [CANDIDATE]: Term 1 election: Requested vote from peers ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1:38801), db54b07b493c45bd911b6138868a7513 (127.0.97.3:33485)
I20260926 02:59:31.423445 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd146feb1a41ccad021f7491f93c1b" candidate_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2"
I20260926 02:59:31.423638 1675 raft_consensus.cc:3037] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.423838 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.408683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34567 (local address 127.0.97.3:33485)
0926 02:59:31.408846 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.408852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.413656 (+ 4804us) server_negotiation.cc:407] Connection header received
0926 02:59:31.413828 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.413834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.413906 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.414008 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:31.414654 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.415516 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.417844 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.418988 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.419136 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.419265 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.423422 (+ 4157us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.423567 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 02:59:31.423902 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.408573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33485 (local address 127.0.97.2:34567)
0926 02:59:31.413526 (+ 4953us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.413546 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.413679 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.414043 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.414048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.414073 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:31.414480 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.414493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.415631 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.415639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.417684 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.417696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.419204 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.419210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.423205 (+ 3995us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.423229 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.423391 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.423524 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4822,"mutex_wait_us":22,"thread_start_us":4630,"threads_started":1}
I20260926 02:59:31.424647 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd146feb1a41ccad021f7491f93c1b" candidate_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db54b07b493c45bd911b6138868a7513" is_pre_election: true
I20260926 02:59:31.424647 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd146feb1a41ccad021f7491f93c1b" candidate_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db54b07b493c45bd911b6138868a7513"
I20260926 02:59:31.424888 1948 raft_consensus.cc:2454] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd25fd9fa0d140aa9a58a6a9254a129e in term 0.
I20260926 02:59:31.426283 1675 raft_consensus.cc:2454] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd25fd9fa0d140aa9a58a6a9254a129e in term 1.
I20260926 02:59:31.426697 1741 leader_election.cc:302] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ceafb7ea501f46cbadd3751d45b6d5d2, fd25fd9fa0d140aa9a58a6a9254a129e; no voters: db54b07b493c45bd911b6138868a7513
I20260926 02:59:31.426994 2023 raft_consensus.cc:2781] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:31.427469 2023 raft_consensus.cc:686] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 LEADER]: Becoming Leader. State: Replica: fd25fd9fa0d140aa9a58a6a9254a129e, State: Running, Role: LEADER
I20260926 02:59:31.428022 2023 consensus_queue.cc:237] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } }
I20260926 02:59:31.430202 2060 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: TxnStatusTablet state changed. Reason: New leader fd25fd9fa0d140aa9a58a6a9254a129e. Latest consensus state: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.430569 2060 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:59:31.431736 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:59:31.431947 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:59:31.432168 1512 catalog_manager.cc:5495] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e reported cstate change: term changed from 0 to 1, leader changed from <none> to fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } health_report { overall_health: UNKNOWN } } }
W20260926 02:59:31.449637 1860 tablet.cc:2369] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:59:31.460188 1721 tablet.cc:2369] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:59:31.476864 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.468324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36169 (local address 127.0.0.1:35098)
0926 02:59:31.468766 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.468787 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.468927 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.469919 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.469925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.469938 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:31.470125 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.470138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.471637 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.471642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.473264 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.473278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.474420 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.474426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.475085 (+ 659us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.475154 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.476273 (+ 1119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.476293 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.476373 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.476523 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260926 02:59:31.477670 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.469248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.0.97.2:36169)
0926 02:59:31.469405 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.469411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.469427 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:59:31.469512 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.469518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.469605 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.469751 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:31.470288 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.471442 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.473426 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.474140 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.474284 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.475383 (+ 1099us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.476082 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.476218 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.477320 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.477414 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:59:31.482429 1948 raft_consensus.cc:3037] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.485353 1675 raft_consensus.cc:1261] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Refusing update from remote peer fd25fd9fa0d140aa9a58a6a9254a129e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:31.486089 2060 consensus_queue.cc:1035] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:31.487010 1948 raft_consensus.cc:1261] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513 [term 1 FOLLOWER]: Refusing update from remote peer fd25fd9fa0d140aa9a58a6a9254a129e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:31.487692 2060 consensus_queue.cc:1035] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:31.498595 2024 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: TxnStatusTablet state changed. Reason: New leader fd25fd9fa0d140aa9a58a6a9254a129e. Latest consensus state: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.498934 2024 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:31.498788 2025 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: TxnStatusTablet state changed. Reason: New leader fd25fd9fa0d140aa9a58a6a9254a129e. Latest consensus state: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.499092 2025 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:31.500768 2060 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.502271 2060 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:59:31.504652 2025 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.504933 2025 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P ceafb7ea501f46cbadd3751d45b6d5d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:31.506705 2060 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.506915 2060 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P fd25fd9fa0d140aa9a58a6a9254a129e: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:59:31.509573 2024 tablet_replica.cc:439] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceafb7ea501f46cbadd3751d45b6d5d2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38801 } } peers { permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 } } peers { permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 } } }
I20260926 02:59:31.509860 2024 tablet_replica.cc:442] T d5dd146feb1a41ccad021f7491f93c1b P db54b07b493c45bd911b6138868a7513: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:31.516534 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.508783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.0.97.1:38801)
0926 02:59:31.508929 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.508934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.508950 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:59:31.509307 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.509313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.509419 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.509576 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:31.511462 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.512595 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.514098 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.514662 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.514789 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.515219 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.515820 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.515956 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.516032 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.516105 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 02:59:31.532266 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.526263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53501 (local address 127.0.97.2:36169)
0926 02:59:31.526688 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.526694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.526997 (+ 303us) server_negotiation.cc:407] Connection header received
0926 02:59:31.527188 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.527193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.527267 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.527359 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:31.527806 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.528553 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.530089 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.530908 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.531051 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.531484 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.531584 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.531734 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.531916 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.531987 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260926 02:59:31.532362 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.526262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36169 (local address 127.0.97.1:53501)
0926 02:59:31.526912 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.526932 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.527058 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.527383 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.527388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.527398 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:31.527661 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.527668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.528678 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.528686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.529954 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.529965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.531081 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.531088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.531224 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.531277 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.531727 (+ 450us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.531737 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.531836 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.531962 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"spinlock_wait_cycles":1408,"thread_start_us":464,"threads_started":1}
I20260926 02:59:31.542090 1947 raft_consensus.cc:3037] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:31.543226 1814 raft_consensus.cc:1261] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Refusing update from remote peer ceafb7ea501f46cbadd3751d45b6d5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:31.544322 2025 consensus_queue.cc:1035] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd25fd9fa0d140aa9a58a6a9254a129e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 02:59:31.546929 1947 raft_consensus.cc:1261] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 1 FOLLOWER]: Refusing update from remote peer ceafb7ea501f46cbadd3751d45b6d5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:31.547667 2025 consensus_queue.cc:1035] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db54b07b493c45bd911b6138868a7513" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:31.613529 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.597812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54649 (local address 127.0.97.1:38801)
0926 02:59:31.598018 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.598026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.605171 (+ 7145us) server_negotiation.cc:407] Connection header received
0926 02:59:31.605604 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.605612 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.605692 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.605757 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:31.607370 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.608067 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.611336 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.611977 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.612048 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.612796 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.612918 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.613044 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.613184 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.613256 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:59:31.614029 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.597329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38801 (local address 127.0.97.2:54649)
0926 02:59:31.605066 (+ 7737us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:31.605093 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:31.605437 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:31.606441 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:31.606446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:31.606458 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:31.607131 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.607147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.608191 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.608199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.611157 (+ 2958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:31.611170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.612376 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.612382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:31.612549 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.612605 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:31.613042 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:31.613054 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:59:31.613149 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 02:59:31.613292 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6940,"thread_start_us":6796,"threads_started":1}
I20260926 02:59:31.669220 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:31.669488 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:31.672884 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a597af5eeddd42a7b0febc2dd32fb907: Sending DeleteTablet for 3 replicas of tablet 80148913ff95455996a63ab2ba112399
I20260926 02:59:31.674118 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 80148913ff95455996a63ab2ba112399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35082
I20260926 02:59:31.674434 2040 tablet_replica.cc:330] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: stopping tablet replica
I20260926 02:59:31.674651 2040 raft_consensus.cc:2229] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:31.674705 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 80148913ff95455996a63ab2ba112399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58506
I20260926 02:59:31.674837 2040 raft_consensus.cc:2258] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:31.674942 2041 tablet_replica.cc:330] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: stopping tablet replica
I20260926 02:59:31.675114 2041 raft_consensus.cc:2229] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:31.675557 2041 raft_consensus.cc:2258] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:31.676100 2040 ts_tablet_manager.cc:1905] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:31.676860 2041 ts_tablet_manager.cc:1905] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:31.684006 2041 ts_tablet_manager.cc:1918] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:31.684264 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 80148913ff95455996a63ab2ba112399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39470
I20260926 02:59:31.684559 2039 tablet_replica.cc:330] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: stopping tablet replica
I20260926 02:59:31.684875 2041 log.cc:1196] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/80148913ff95455996a63ab2ba112399
I20260926 02:59:31.685415 2041 ts_tablet_manager.cc:1939] T 80148913ff95455996a63ab2ba112399 P ceafb7ea501f46cbadd3751d45b6d5d2: Deleting consensus metadata
I20260926 02:59:31.685374 2039 raft_consensus.cc:2229] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:31.685681 2039 raft_consensus.cc:2258] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:31.687011 2039 ts_tablet_manager.cc:1905] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:31.689272 2040 ts_tablet_manager.cc:1918] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:31.689419 2040 log.cc:1196] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/80148913ff95455996a63ab2ba112399
I20260926 02:59:31.690945 2040 ts_tablet_manager.cc:1939] T 80148913ff95455996a63ab2ba112399 P fd25fd9fa0d140aa9a58a6a9254a129e: Deleting consensus metadata
I20260926 02:59:31.692281 1498 catalog_manager.cc:4841] TS fd25fd9fa0d140aa9a58a6a9254a129e (127.0.97.2:36169): tablet 80148913ff95455996a63ab2ba112399 (table test-table [id=cefb0a6e33b94117a5571891c693f995]) successfully deleted
I20260926 02:59:31.694309 2039 ts_tablet_manager.cc:1918] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:31.694576 2039 log.cc:1196] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/80148913ff95455996a63ab2ba112399
I20260926 02:59:31.695098 2039 ts_tablet_manager.cc:1939] T 80148913ff95455996a63ab2ba112399 P db54b07b493c45bd911b6138868a7513: Deleting consensus metadata
I20260926 02:59:31.706404 1498 catalog_manager.cc:4841] TS ceafb7ea501f46cbadd3751d45b6d5d2 (127.0.97.1:38801): tablet 80148913ff95455996a63ab2ba112399 (table test-table [id=cefb0a6e33b94117a5571891c693f995]) successfully deleted
I20260926 02:59:31.707134 1498 catalog_manager.cc:4841] TS db54b07b493c45bd911b6138868a7513 (127.0.97.3:33485): tablet 80148913ff95455996a63ab2ba112399 (table test-table [id=cefb0a6e33b94117a5571891c693f995]) successfully deleted
Sep 26 02:59:31 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:31.726564 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.688778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.0.97.1:38801)
0926 02:59:31.688958 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.688964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.705517 (+ 16553us) server_negotiation.cc:407] Connection header received
0926 02:59:31.705836 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.705842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.705928 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.706003 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:31.708916 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.710062 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.711888 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.712544 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.712646 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.718253 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:31.718267 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:31.718287 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:31.718334 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:31.722798 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.723692 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.723707 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.723711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.723851 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.724286 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.724291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.724295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.724737 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:31.725085 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.725720 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.725955 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 02:59:31.728565 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58530
I20260926 02:59:31.765377 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.742343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.0.97.62:37683)
0926 02:59:31.742558 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.742565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.749340 (+ 6775us) server_negotiation.cc:407] Connection header received
0926 02:59:31.749734 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.749740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.749825 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.749898 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:31.751746 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.752917 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.754782 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.755379 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.755471 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.758804 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:31.758810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:31.758814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:31.758850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:31.762340 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.763139 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.763145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.763148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.763292 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.763713 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.763718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.763722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.764044 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:31.764271 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.764785 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.764988 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 02:59:31.766932 1512 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56380
I20260926 02:59:31.769992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1584
I20260926 02:59:31.802253 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.772569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58542 (local address 127.0.97.1:38801)
0926 02:59:31.772733 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.772740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.781119 (+ 8379us) server_negotiation.cc:407] Connection header received
0926 02:59:31.781552 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.781563 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.781649 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.781719 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:31.783474 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.784657 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.786515 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.787204 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.787303 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.795633 (+ 8330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:31.795639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:31.795644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:31.795688 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:31.799152 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.799896 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.799903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.799907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.800031 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.800463 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.800468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.800471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.800838 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:31.801124 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.801644 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.801872 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 02:59:31.803748 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58542
W20260926 02:59:31.804373 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:31.804535 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 1584
I20260926 02:59:31.824033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1724
Sep 26 02:59:31 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 02:59:31.839320 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.824810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.0.97.2:36169)
0926 02:59:31.824969 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.824973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.825056 (+ 83us) server_negotiation.cc:407] Connection header received
0926 02:59:31.825215 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.825219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.825289 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.825372 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:31.826859 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.827562 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.828778 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.829304 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.829489 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.833344 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:31.833354 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:31.833370 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:31.833409 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:31.836634 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.837363 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.837370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.837372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.837486 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.837850 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.837855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.837858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.838243 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:31.838551 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.838777 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.838978 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 02:59:31.840046 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35100
W20260926 02:59:31.840705 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:31.840893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 1724
I20260926 02:59:31.862125 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1863
Sep 26 02:59:31 dist-test-slave-fccw krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391570, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 02:59:31.875119 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:31.862814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.0.97.3:33485)
0926 02:59:31.863000 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:31.863004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:31.863020 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:59:31.863157 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:31.863160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:31.863217 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:31.863312 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:31.864384 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.865125 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.866200 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:31.866560 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:31.866643 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:31.870188 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:31.870200 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:31.870215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:31.870242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:31.872935 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.873611 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.873617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.873619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.873718 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:31.873987 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:31.873989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:31.873991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:31.874276 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:31.874478 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:31.874636 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 02:59:31.874777 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 02:59:31.875674 1993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39486
W20260926 02:59:31.876245 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:31.876355 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 1863
I20260926 02:59:31.897087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 1481
I20260926 02:59:31.898120 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:32.007944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 1481
2026-09-26T02:59:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4317 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:32 dist-test-slave-fccw krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:32 dist-test-slave-fccw krb5kdc[2087](info): set up 2 sockets
Sep 26 02:59:32 dist-test-slave-fccw krb5kdc[2087](info): commencing operation
krb5kdc: starting...
W20260926 02:59:33.065097 388 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:33 dist-test-slave-fccw krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:33.242653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43715
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:33.544101 2100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:33.544371 2100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:33.544456 2100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:33.551183 2100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:33.551273 2100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 02:59:33.551364 2100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:33.551460 2100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:33.551537 2100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:33.551622 2100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:33.560933 2100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:33.562193 2100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:33.563508 2100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:33.575212 2106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:33.575161 2108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:33.575624 2105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:33.576200 2100 server_base.cc:1034] running on GCE node
I20260926 02:59:33.576889 2100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:33.578182 2100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:33.579411 2100 hybrid_clock.cc:648] HybridClock initialized: now 1790391573579354 us; error 40 us; skew 500 ppm
I20260926 02:59:33.579713 2100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:33 dist-test-slave-fccw krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:33.582792 2100 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:33.584165 2100 webserver.cc:460] Webserver started at http://127.0.97.62:46537/ using document root <none> and password file <none>
I20260926 02:59:33.584861 2100 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:33.584985 2100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:33.585332 2100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:33.588048 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "5d54a736388d47eaae8103af55dde138"
format_stamp: "Formatted at 2026-09-26 02:59:33 on dist-test-slave-fccw"
server_key: "f58768257544c40552de6378e0057c6d"
server_key_iv: "d9879ff316db8398442a81c733eb64a2"
server_key_version: "encryptionkey@0"
I20260926 02:59:33.588855 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d54a736388d47eaae8103af55dde138"
format_stamp: "Formatted at 2026-09-26 02:59:33 on dist-test-slave-fccw"
server_key: "f58768257544c40552de6378e0057c6d"
server_key_iv: "d9879ff316db8398442a81c733eb64a2"
server_key_version: "encryptionkey@0"
I20260926 02:59:33.593573 2100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 02:59:33.596704 2115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:33.597946 2100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 02:59:33.598107 2100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "5d54a736388d47eaae8103af55dde138"
format_stamp: "Formatted at 2026-09-26 02:59:33 on dist-test-slave-fccw"
server_key: "f58768257544c40552de6378e0057c6d"
server_key_iv: "d9879ff316db8398442a81c733eb64a2"
server_key_version: "encryptionkey@0"
I20260926 02:59:33.598306 2100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:33.611976 2100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:33.615976 2100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:33.616238 2100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:33.636567 2100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43715
I20260926 02:59:33.636587 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43715 every 8 connection(s)
I20260926 02:59:33.637974 2100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:33.641573 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:33.648219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 2100
I20260926 02:59:33.648434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:33.649077 388 external_mini_cluster.cc:1428] Setting key dfad420f5f6eee2f78f44952ca2f5647
I20260926 02:59:33.651654 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Bootstrap starting.
I20260926 02:59:33.654911 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:33.656179 2177 log.cc:824] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:33.659163 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: No bootstrap required, opened a new log
Sep 26 02:59:33 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:33.667913 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d54a736388d47eaae8103af55dde138" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43715 } }
I20260926 02:59:33.668255 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:33.668399 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d54a736388d47eaae8103af55dde138, State: Initialized, Role: FOLLOWER
I20260926 02:59:33.668906 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d54a736388d47eaae8103af55dde138" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43715 } }
I20260926 02:59:33.669212 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:33.669354 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:33.669523 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:33.671500 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:33.651008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.0.97.62:43715)
0926 02:59:33.651826 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:33.651837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:33.651902 (+ 65us) server_negotiation.cc:407] Connection header received
0926 02:59:33.653651 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:33.653672 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:33.654007 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:33.654313 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:33.656064 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:33.657789 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:33.659491 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:33.660132 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:33.660258 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:33.664656 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:33.664663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:33.664680 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:33.664710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:33.667601 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:33.668233 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:33.668239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:33.668248 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:33.668434 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:33.668876 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:33.668881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:33.668884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:33.669545 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:33.669841 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:33.670207 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 02:59:33.670607 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":374,"threads_started":1}
I20260926 02:59:33.671829 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d54a736388d47eaae8103af55dde138" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43715 } }
I20260926 02:59:33.672312 2177 leader_election.cc:302] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d54a736388d47eaae8103af55dde138; no voters:
I20260926 02:59:33.672887 2177 leader_election.cc:288] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:33.673055 2182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:33.673692 2182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [term 1 LEADER]: Becoming Leader. State: Replica: 5d54a736388d47eaae8103af55dde138, State: Running, Role: LEADER
I20260926 02:59:33.674198 2182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d54a736388d47eaae8103af55dde138" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43715 } }
I20260926 02:59:33.676155 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d54a736388d47eaae8103af55dde138. Latest consensus state: current_term: 1 leader_uuid: "5d54a736388d47eaae8103af55dde138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d54a736388d47eaae8103af55dde138" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43715 } } }
I20260926 02:59:33.676445 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:33.676781 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d54a736388d47eaae8103af55dde138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d54a736388d47eaae8103af55dde138" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43715 } } }
I20260926 02:59:33.677182 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:33.677641 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:33.679404 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:33.683722 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:33.691478 2186 catalog_manager.cc:1347] Generated new cluster ID: f594895f1b83473789f2371ad1f48d3b
I20260926 02:59:33.691576 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:33.707026 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:33.708652 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:33.716848 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Generated new TSK 0
I20260926 02:59:33.717736 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:33.768496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43715
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:34.099303 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:34.099597 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:34.099761 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:34.106954 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:34.107086 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:34.107240 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:59:34.107322 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:59:34.107453 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:34.116739 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:34.117985 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:34.119371 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:34.131721 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:34.131795 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:34.132036 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:34.132707 2203 server_base.cc:1034] running on GCE node
I20260926 02:59:34.133342 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:34.134172 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:34.135352 2203 hybrid_clock.cc:648] HybridClock initialized: now 1790391574135321 us; error 32 us; skew 500 ppm
I20260926 02:59:34.135679 2203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:34 dist-test-slave-fccw krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:34.138574 2203 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:34.139853 2203 webserver.cc:460] Webserver started at http://127.0.97.1:34197/ using document root <none> and password file <none>
I20260926 02:59:34.140511 2203 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:34.140650 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:34.140938 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:34.143734 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "39fc761fbe764257a2d0466ea73d3bdd"
format_stamp: "Formatted at 2026-09-26 02:59:34 on dist-test-slave-fccw"
server_key: "25e6e39131220a821255511049beb9c8"
server_key_iv: "8fb8657c68edc603bef0a61feda2f7f2"
server_key_version: "encryptionkey@0"
I20260926 02:59:34.144505 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "39fc761fbe764257a2d0466ea73d3bdd"
format_stamp: "Formatted at 2026-09-26 02:59:34 on dist-test-slave-fccw"
server_key: "25e6e39131220a821255511049beb9c8"
server_key_iv: "8fb8657c68edc603bef0a61feda2f7f2"
server_key_version: "encryptionkey@0"
I20260926 02:59:34.149225 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 02:59:34.152348 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:34.153596 2203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:34.153767 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "39fc761fbe764257a2d0466ea73d3bdd"
format_stamp: "Formatted at 2026-09-26 02:59:34 on dist-test-slave-fccw"
server_key: "25e6e39131220a821255511049beb9c8"
server_key_iv: "8fb8657c68edc603bef0a61feda2f7f2"
server_key_version: "encryptionkey@0"
I20260926 02:59:34.153966 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:34.165256 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:34.169250 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:34.169515 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:34.172024 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:34.172154 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:34.172276 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:34.172382 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:59:34 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:34.195643 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.172542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37637 (local address 127.0.97.62:43715)
0926 02:59:34.172783 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:34.172793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:34.174150 (+ 1357us) server_negotiation.cc:407] Connection header received
0926 02:59:34.177133 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:34.177139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:34.177214 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:34.177332 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:34.179550 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.180838 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.183433 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.184004 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.184136 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.188740 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:34.188745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:34.188749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:34.188785 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:34.192134 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.193174 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.193187 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.193193 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.193362 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.193999 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.194004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.194007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.194304 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:34.194455 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:34.195102 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 02:59:34.195302 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:59:34.196645 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.173168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43715 (local address 127.0.97.1:37637)
0926 02:59:34.173965 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:34.174023 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:34.176801 (+ 2778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:34.177637 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:34.177650 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:34.178427 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:34.179341 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.179374 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.181018 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.181029 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.183299 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.183309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.184246 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.184256 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.184451 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.185217 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:34.185252 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:34.188521 (+ 3269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:34.192279 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.192288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.192304 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.192985 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.193524 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.193528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.193537 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.193837 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.194435 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:34.194442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:34.194762 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 02:59:34.195016 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 02:59:34.195496 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":328,"threads_started":1}
I20260926 02:59:34.217670 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.203063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42995 (local address 127.0.97.62:43715)
0926 02:59:34.203206 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:34.203210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:34.203895 (+ 685us) server_negotiation.cc:407] Connection header received
0926 02:59:34.204097 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:34.204100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:34.204153 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:34.204247 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:34.205119 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.206019 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.207526 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.208133 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.208270 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.210734 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:34.210743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:34.210750 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:34.210794 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:34.214439 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.215317 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.215323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.215327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.215489 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.215930 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.215938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.215941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.216359 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:34.216510 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:34.216953 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 02:59:34.217344 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 02:59:34.217726 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.203162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43715 (local address 127.0.97.1:42995)
0926 02:59:34.203796 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:34.203811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:34.203963 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:34.204268 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:34.204272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:34.204702 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:34.204951 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.204959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.206215 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.206219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.207366 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.207377 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.208363 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.208370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.208528 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.209368 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:34.209401 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:34.210521 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:34.214650 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.214658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.214664 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.215180 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.215633 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.215640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.215649 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.215812 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.216487 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:34.216492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:34.216700 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 02:59:34.216856 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 02:59:34.217230 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":362,"threads_started":1}
I20260926 02:59:34.223886 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37215
I20260926 02:59:34.223918 2337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37215 every 8 connection(s)
I20260926 02:59:34.224969 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:34.226768 2338 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43715
I20260926 02:59:34.226799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 2203
I20260926 02:59:34.226981 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:34.227001 2338 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:34.227720 2338 heartbeater.cc:511] Master 127.0.97.62:43715 requested a full tablet report, sending...
I20260926 02:59:34.227761 388 external_mini_cluster.cc:1428] Setting key 0fccc9bb1b0820a8387f7b3a639493e2
I20260926 02:59:34.230043 2132 ts_manager.cc:194] Registered new tserver with Master: 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215)
I20260926 02:59:34.231392 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37637
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:34.265468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43715
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:34.583776 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:34.584024 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:34.584218 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:34.591130 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:34.591256 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:34.591408 2342 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:59:34.591486 2342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:59:34.591598 2342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:34.600935 2342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:34.602244 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:34.603555 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:34.615722 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:34.615761 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:34.615787 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:34.616564 2342 server_base.cc:1034] running on GCE node
I20260926 02:59:34.617193 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:34.618067 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:34.619289 2342 hybrid_clock.cc:648] HybridClock initialized: now 1790391574619246 us; error 56 us; skew 500 ppm
I20260926 02:59:34.619592 2342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:34 dist-test-slave-fccw krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:34.622694 2342 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:34.623952 2342 webserver.cc:460] Webserver started at http://127.0.97.2:35633/ using document root <none> and password file <none>
I20260926 02:59:34.624621 2342 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:34.624758 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:34.625119 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:34.627743 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "0121773dadc34b1eb8f0cd3325f0ff04"
format_stamp: "Formatted at 2026-09-26 02:59:34 on dist-test-slave-fccw"
server_key: "ec80311ebee92b52d8ae695648334f2d"
server_key_iv: "e7fda19f9c20364fe4ab26202e0a9662"
server_key_version: "encryptionkey@0"
I20260926 02:59:34.628485 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0121773dadc34b1eb8f0cd3325f0ff04"
format_stamp: "Formatted at 2026-09-26 02:59:34 on dist-test-slave-fccw"
server_key: "ec80311ebee92b52d8ae695648334f2d"
server_key_iv: "e7fda19f9c20364fe4ab26202e0a9662"
server_key_version: "encryptionkey@0"
I20260926 02:59:34.633496 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 02:59:34.636978 2357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:34.638279 2342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:59:34.638441 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "0121773dadc34b1eb8f0cd3325f0ff04"
format_stamp: "Formatted at 2026-09-26 02:59:34 on dist-test-slave-fccw"
server_key: "ec80311ebee92b52d8ae695648334f2d"
server_key_iv: "e7fda19f9c20364fe4ab26202e0a9662"
server_key_version: "encryptionkey@0"
I20260926 02:59:34.638617 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:34.667268 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:34.670868 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:34.671149 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:34.673519 2342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:34.673650 2342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:34.673765 2342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:34.673851 2342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:59:34 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:34.695576 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.674298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36637 (local address 127.0.97.62:43715)
0926 02:59:34.674487 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:34.674495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:34.675905 (+ 1410us) server_negotiation.cc:407] Connection header received
0926 02:59:34.678008 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:34.678014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:34.678097 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:34.678224 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:34.680384 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.681552 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.684109 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.684791 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.684942 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.689053 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:34.689059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:34.689064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:34.689124 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:34.692631 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.693464 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.693471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.693475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.693593 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.694003 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.694008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.694011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.694303 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:34.694444 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:34.695062 (+ 618us) server_negotiation.cc:299] Negotiation successful
0926 02:59:34.695276 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 02:59:34.696734 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.674967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43715 (local address 127.0.97.2:36637)
0926 02:59:34.675756 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:34.675799 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:34.677752 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:34.678483 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:34.678493 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:34.679255 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:34.680144 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.680175 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.681751 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.681761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.683933 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.683966 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.685070 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.685081 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.685243 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.685859 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:34.685879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:34.688726 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:34.692781 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.692792 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.692810 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.693342 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.693706 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.693712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.693716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.693894 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.694441 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:34.694449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:34.694729 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 02:59:34.694967 (+ 238us) client_negotiation.cc:240] Negotiation successful
0926 02:59:34.695419 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":386,"threads_started":1}
I20260926 02:59:34.717181 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.702434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38881 (local address 127.0.97.62:43715)
0926 02:59:34.702585 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:34.702589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:34.703281 (+ 692us) server_negotiation.cc:407] Connection header received
0926 02:59:34.703462 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:34.703467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:34.703536 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:34.703646 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:34.705160 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.705886 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.708324 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.708703 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.708791 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.711383 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:34.711388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:34.711391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:34.711417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:34.714341 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.715228 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.715231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.715234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.715342 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:34.715763 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:34.715766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:34.715768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:34.716012 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:34.716111 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:34.716592 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 02:59:34.716740 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 02:59:34.717767 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:34.702590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43715 (local address 127.0.97.2:38881)
0926 02:59:34.703184 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:34.703200 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:34.703348 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:34.703691 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:34.703699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:34.704410 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:34.705009 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.705022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.706027 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.706034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.708181 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:34.708198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:34.708897 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:34.708906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:34.709195 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:34.709888 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:34.709917 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:34.711207 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:34.714460 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.714468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.714474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.715111 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.715457 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:34.715464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:34.715470 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:34.715653 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:34.716118 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:34.716126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:34.716356 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 02:59:34.716531 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 02:59:34.717061 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":360,"threads_started":1}
I20260926 02:59:34.729586 2342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45561
I20260926 02:59:34.729624 2476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45561 every 8 connection(s)
I20260926 02:59:34.730772 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:34.732151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 2342
I20260926 02:59:34.732345 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:34.732828 388 external_mini_cluster.cc:1428] Setting key c6aa1b3494c30178f284437c62196507
I20260926 02:59:34.732812 2477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43715
I20260926 02:59:34.733495 2477 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:34.734257 2477 heartbeater.cc:511] Master 127.0.97.62:43715 requested a full tablet report, sending...
I20260926 02:59:34.735577 2132 ts_manager.cc:194] Registered new tserver with Master: 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561)
I20260926 02:59:34.736356 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36637
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:34.771489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43715
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:35.076345 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:35.076630 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:35.076798 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:35.083940 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:35.084079 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:35.084224 2481 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 02:59:35.084309 2481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 02:59:35.084435 2481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:35.095134 2481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:43715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:35.096695 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:35.098589 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:35.111581 2486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:35.111644 2487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:35.111594 2489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:35.112394 2481 server_base.cc:1034] running on GCE node
I20260926 02:59:35.112934 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:35.113807 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:35.115041 2481 hybrid_clock.cc:648] HybridClock initialized: now 1790391575114991 us; error 56 us; skew 500 ppm
I20260926 02:59:35.115356 2481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:35 dist-test-slave-fccw krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:35.118328 2481 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:35.119686 2481 webserver.cc:460] Webserver started at http://127.0.97.3:44351/ using document root <none> and password file <none>
I20260926 02:59:35.120329 2481 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:35.120483 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:35.120786 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:35.123539 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "271ea220deeb492c8ad048b00492469f"
format_stamp: "Formatted at 2026-09-26 02:59:35 on dist-test-slave-fccw"
server_key: "499f7bc4d61e2e8e6726d6ab024fb87f"
server_key_iv: "47620a2e119a6f2ff87843d8bd6eb473"
server_key_version: "encryptionkey@0"
I20260926 02:59:35.124290 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "271ea220deeb492c8ad048b00492469f"
format_stamp: "Formatted at 2026-09-26 02:59:35 on dist-test-slave-fccw"
server_key: "499f7bc4d61e2e8e6726d6ab024fb87f"
server_key_iv: "47620a2e119a6f2ff87843d8bd6eb473"
server_key_version: "encryptionkey@0"
I20260926 02:59:35.129076 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 02:59:35.132105 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:35.133302 2481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:35.133459 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "271ea220deeb492c8ad048b00492469f"
format_stamp: "Formatted at 2026-09-26 02:59:35 on dist-test-slave-fccw"
server_key: "499f7bc4d61e2e8e6726d6ab024fb87f"
server_key_iv: "47620a2e119a6f2ff87843d8bd6eb473"
server_key_version: "encryptionkey@0"
I20260926 02:59:35.133642 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:35.150017 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:35.154052 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:35.154341 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:35.156805 2481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:35.156948 2481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:35.157096 2481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:35.157200 2481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 02:59:35 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:35.179960 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.157484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42997 (local address 127.0.97.62:43715)
0926 02:59:35.157688 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.157694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.159065 (+ 1371us) server_negotiation.cc:407] Connection header received
0926 02:59:35.161031 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.161040 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.161145 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.161263 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:35.163300 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.164577 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.167194 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.167797 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.167930 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.172378 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:35.172387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:35.172395 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:35.172447 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:35.176512 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.177493 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.177503 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.177509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.177709 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.178137 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.178144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.178150 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.178566 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:35.178752 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.179388 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.179607 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 02:59:35.180910 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.157996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43715 (local address 127.0.97.3:42997)
0926 02:59:35.158888 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.158937 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.160740 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.161517 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.161527 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.162263 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:35.163079 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.163116 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.164746 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.164755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.167053 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.167065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.168032 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.168039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.168227 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.168940 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:35.168980 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:35.171992 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:35.176679 (+ 4687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:35.176693 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:35.176714 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:35.177365 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:35.177825 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:35.177829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:35.177831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:35.178014 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:35.178723 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:35.178728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:35.179026 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.179290 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.179791 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":414,"threads_started":1}
I20260926 02:59:35.202368 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.186717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43715 (local address 127.0.97.3:49645)
0926 02:59:35.187420 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.187444 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.187610 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.187945 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.187952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.188696 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:35.189191 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.189207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.190173 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.190182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.192331 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.192349 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.193119 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.193129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.193361 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.194170 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:35.194187 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:35.195292 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:35.199566 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:35.199577 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:35.199589 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:35.199991 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:35.200511 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:35.200515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:35.200517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:35.200637 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:35.201391 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:35.201394 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:35.201536 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.201670 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.201989 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":406,"threads_started":1}
I20260926 02:59:35.202605 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.186601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49645 (local address 127.0.97.62:43715)
0926 02:59:35.186766 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.186773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.187543 (+ 770us) server_negotiation.cc:407] Connection header received
0926 02:59:35.187735 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.187738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.187795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.187871 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:35.189375 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.190025 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.192477 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.192896 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.192986 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.195498 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:35.195503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:35.195506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:35.195533 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:35.199434 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.200146 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.200155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.200158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.200383 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.200763 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.200771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.200777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.201271 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:35.201465 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.201807 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.202093 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:59:35.212759 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45453
I20260926 02:59:35.212792 2615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45453 every 8 connection(s)
I20260926 02:59:35.213979 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:35.216013 2616 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43715
I20260926 02:59:35.216238 2616 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:35.216843 2616 heartbeater.cc:511] Master 127.0.97.62:43715 requested a full tablet report, sending...
I20260926 02:59:35.216925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 2481
I20260926 02:59:35.217124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:35.217523 388 external_mini_cluster.cc:1428] Setting key 63b551eefc3404a44d0cfc8128659255
I20260926 02:59:35.218240 2132 ts_manager.cc:194] Registered new tserver with Master: 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:35.219225 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42997
I20260926 02:59:35.220368 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:59:35.234727 2338 heartbeater.cc:503] Master 127.0.97.62:43715 was elected leader, sending a full tablet report...
Sep 26 02:59:35 dist-test-slave-fccw krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:59:35 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:35.265803 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.253024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.0.97.62:43715)
0926 02:59:35.253245 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.253254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.253487 (+ 233us) server_negotiation.cc:407] Connection header received
0926 02:59:35.253700 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.253704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.253772 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.253877 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:35.255173 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.255866 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.257114 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.257560 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.257702 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.261414 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:35.261418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:35.261422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:35.261456 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:35.263638 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.264194 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.264197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.264200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.264311 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.264624 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.264626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.264628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.264860 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:35.265102 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.265313 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.265462 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:59:35.269429 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:59:35.276028 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:35.298071 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.288252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45561 (local address 127.0.0.1:56514)
0926 02:59:35.289594 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.289614 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.289726 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.290513 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.290520 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.290546 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:35.291030 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.291046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.293033 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.293038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.295354 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.295371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.297150 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.297157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.297351 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.297413 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.297529 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.297704 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"thread_start_us":1099,"threads_started":1}
I20260926 02:59:35.299407 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.289347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.97.2:45561)
0926 02:59:35.289867 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.289876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.289904 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:59:35.290010 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.290021 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.290286 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.291249 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:35.291430 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.292859 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.295562 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.296897 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.298599 (+ 1702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.298859 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.298926 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.299076 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":390,"threads_started":1}
I20260926 02:59:35.300647 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.288204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45453 (local address 127.0.0.1:46968)
0926 02:59:35.288825 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.288869 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.289053 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.290914 (+ 1861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.290924 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.290956 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:35.291357 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.291368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.294439 (+ 3071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.294444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.296599 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.296612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.299789 (+ 3177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.299795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.300006 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.300058 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.300169 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.300304 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":424,"threads_started":1}
I20260926 02:59:35.301091 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.288688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.97.3:45453)
0926 02:59:35.290346 (+ 1658us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.290354 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.290385 (+ 31us) server_negotiation.cc:407] Connection header received
0926 02:59:35.290479 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.290486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.290705 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.291542 (+ 837us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:35.292688 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.294296 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.297778 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.299478 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.299733 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.299940 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.300601 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.300756 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":1341,"threads_started":1}
I20260926 02:59:35.304040 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 43f20141c8fc4bf68d263eefd9dd3d78 (DEFAULT_TABLE table=test-table [id=47d0d483c7cb4d9db513d34cbd7fd9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:35.304433 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 43f20141c8fc4bf68d263eefd9dd3d78 (DEFAULT_TABLE table=test-table [id=47d0d483c7cb4d9db513d34cbd7fd9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:35.305472 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f20141c8fc4bf68d263eefd9dd3d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.305723 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f20141c8fc4bf68d263eefd9dd3d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.308108 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.289713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37215 (local address 127.0.0.1:43652)
0926 02:59:35.298313 (+ 8600us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.298330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.298463 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.302370 (+ 3907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.302374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.302396 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:35.302543 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.302558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.303841 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.303849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.305098 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.305109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.307338 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.307343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.307520 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.307563 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.307667 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.307814 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8512,"thread_start_us":10419,"threads_started":1}
I20260926 02:59:35.308182 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.290111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.0.97.1:37215)
0926 02:59:35.300364 (+ 10253us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.300371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.300397 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:59:35.300519 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.300526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.302176 (+ 1650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.302368 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:35.302686 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.303712 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.305227 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.307187 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.307317 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.307503 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.307726 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.307878 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10099,"thread_start_us":9979,"threads_started":1}
I20260926 02:59:35.310962 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 43f20141c8fc4bf68d263eefd9dd3d78 (DEFAULT_TABLE table=test-table [id=47d0d483c7cb4d9db513d34cbd7fd9cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:35.312093 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43f20141c8fc4bf68d263eefd9dd3d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.320290 2636 tablet_bootstrap.cc:492] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Bootstrap starting.
I20260926 02:59:35.321046 2635 tablet_bootstrap.cc:492] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Bootstrap starting.
I20260926 02:59:35.322894 2637 tablet_bootstrap.cc:492] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Bootstrap starting.
I20260926 02:59:35.324333 2636 tablet_bootstrap.cc:654] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.324720 2635 tablet_bootstrap.cc:654] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.325403 2636 log.cc:824] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:35.325899 2635 log.cc:824] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:35.325973 2637 tablet_bootstrap.cc:654] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.327178 2637 log.cc:824] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:35.333082 2636 tablet_bootstrap.cc:492] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: No bootstrap required, opened a new log
I20260926 02:59:35.333321 2637 tablet_bootstrap.cc:492] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: No bootstrap required, opened a new log
I20260926 02:59:35.333348 2635 tablet_bootstrap.cc:492] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: No bootstrap required, opened a new log
I20260926 02:59:35.333350 2636 ts_tablet_manager.cc:1397] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260926 02:59:35.333567 2635 ts_tablet_manager.cc:1397] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260926 02:59:35.333568 2637 ts_tablet_manager.cc:1397] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260926 02:59:35.340947 2636 raft_consensus.cc:348] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.341370 2636 raft_consensus.cc:374] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.341473 2636 raft_consensus.cc:729] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0121773dadc34b1eb8f0cd3325f0ff04, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.342149 2636 consensus_queue.cc:260] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.344005 2637 raft_consensus.cc:348] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.344005 2635 raft_consensus.cc:348] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.344406 2637 raft_consensus.cc:374] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.344406 2635 raft_consensus.cc:374] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.344509 2635 raft_consensus.cc:729] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271ea220deeb492c8ad048b00492469f, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.344506 2637 raft_consensus.cc:729] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fc761fbe764257a2d0466ea73d3bdd, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.345098 2637 consensus_queue.cc:260] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.345098 2635 consensus_queue.cc:260] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.349275 2616 heartbeater.cc:503] Master 127.0.97.62:43715 was elected leader, sending a full tablet report...
I20260926 02:59:35.349957 2635 ts_tablet_manager.cc:1428] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 02:59:35.349906 2637 ts_tablet_manager.cc:1428] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260926 02:59:35.350682 2477 heartbeater.cc:503] Master 127.0.97.62:43715 was elected leader, sending a full tablet report...
I20260926 02:59:35.351346 2636 ts_tablet_manager.cc:1428] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent starting tablet: real 0.018s user 0.006s sys 0.011s
I20260926 02:59:35.451537 2642 raft_consensus.cc:482] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:35.451859 2642 raft_consensus.cc:504] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.453624 2642 leader_election.cc:288] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453), 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561)
I20260926 02:59:35.460191 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.453967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50411 (local address 127.0.97.3:45453)
0926 02:59:35.454148 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.454154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.454666 (+ 512us) server_negotiation.cc:407] Connection header received
0926 02:59:35.454882 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.454890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.454977 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.455041 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:35.455682 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.456427 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.458501 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.459233 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.459328 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.459449 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.459797 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.459932 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 02:59:35.460232 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.454060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58015 (local address 127.0.97.2:45561)
0926 02:59:35.454246 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.454251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.454818 (+ 567us) server_negotiation.cc:407] Connection header received
0926 02:59:35.454971 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.454975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.455073 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.455170 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:35.455822 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.456550 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.458466 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.459310 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.459453 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.459570 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.459887 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.460014 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 02:59:35.460304 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.453792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45453 (local address 127.0.97.1:50411)
0926 02:59:35.454543 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.454565 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.454719 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.455115 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.455121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.455157 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:35.455536 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.455545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.456575 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.456580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.458332 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.458343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.459444 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.459450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.459646 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.459669 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.459742 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.459870 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":63,"thread_start_us":392,"threads_started":1}
I20260926 02:59:35.460758 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.454023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45561 (local address 127.0.97.1:58015)
0926 02:59:35.454706 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.454725 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.454852 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.455215 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.455218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.455269 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:35.455667 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.455677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.456696 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.456699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.458314 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.458328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.459571 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.459575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.459698 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.459740 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.459843 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.459950 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":3,"spinlock_wait_cycles":8960,"thread_start_us":392,"threads_started":1}
I20260926 02:59:35.461264 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f20141c8fc4bf68d263eefd9dd3d78" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f" is_pre_election: true
I20260926 02:59:35.461848 2571 raft_consensus.cc:2454] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 0.
I20260926 02:59:35.461776 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f20141c8fc4bf68d263eefd9dd3d78" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" is_pre_election: true
I20260926 02:59:35.462291 2432 raft_consensus.cc:2454] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 0.
I20260926 02:59:35.462518 2220 leader_election.cc:302] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271ea220deeb492c8ad048b00492469f, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:35.462924 2642 raft_consensus.cc:2781] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:35.463037 2642 raft_consensus.cc:482] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:35.463155 2642 raft_consensus.cc:3037] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.465337 2642 raft_consensus.cc:504] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
W20260926 02:59:35.465485 2617 tablet.cc:2369] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:59:35.465939 2642 leader_election.cc:288] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 election: Requested vote from peers 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453), 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561)
I20260926 02:59:35.466408 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f20141c8fc4bf68d263eefd9dd3d78" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04"
I20260926 02:59:35.466391 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43f20141c8fc4bf68d263eefd9dd3d78" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f"
I20260926 02:59:35.466600 2571 raft_consensus.cc:3037] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.466590 2432 raft_consensus.cc:3037] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.469775 2571 raft_consensus.cc:2454] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 1.
I20260926 02:59:35.469776 2432 raft_consensus.cc:2454] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 1.
I20260926 02:59:35.470170 2221 leader_election.cc:302] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0121773dadc34b1eb8f0cd3325f0ff04, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:35.470458 2642 raft_consensus.cc:2781] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:35.470882 2642 raft_consensus.cc:686] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 LEADER]: Becoming Leader. State: Replica: 39fc761fbe764257a2d0466ea73d3bdd, State: Running, Role: LEADER
I20260926 02:59:35.471321 2642 consensus_queue.cc:237] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } }
I20260926 02:59:35.475690 2131 catalog_manager.cc:5495] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } health_report { overall_health: HEALTHY } } }
W20260926 02:59:35.477334 2339 tablet.cc:2369] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:59:35.482546 2478 tablet.cc:2369] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 02:59:35 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:35.519327 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.507259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43715 (local address 127.0.0.1:41088)
0926 02:59:35.507805 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.507820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.507937 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.508234 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.508238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.508879 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:35.509229 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.509239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.510184 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.510188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.511100 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.511110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.511775 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.511779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.511894 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.512474 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:35.512490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:35.514589 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:35.517376 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:35.517384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:35.517389 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:35.517746 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:35.518032 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:35.518035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:35.518037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:35.518167 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:35.518575 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:35.518580 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:35.518718 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.518842 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.519024 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":323,"threads_started":1}
I20260926 02:59:35.519409 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.507239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.0.97.62:43715)
0926 02:59:35.507400 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.507405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.507904 (+ 499us) server_negotiation.cc:407] Connection header received
0926 02:59:35.508030 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.508034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.508099 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.508179 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:35.509357 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.510064 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.511219 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.511629 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.511715 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.514763 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:35.514768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:35.514771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:35.514796 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:35.517270 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.517838 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.517841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.517843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.517947 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:35.518249 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:35.518252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:35.518254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:35.518478 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:35.518577 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.518915 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.519129 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":62}
I20260926 02:59:35.524070 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41088:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 02:59:35.525092 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:35.531189 2268 tablet_service.cc:1467] Processing CreateTablet for tablet c534fd8160494e8f9c75852b40da7971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59c55228aa4a48fc8da7b1a2040b18a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:59:35.531277 2407 tablet_service.cc:1467] Processing CreateTablet for tablet c534fd8160494e8f9c75852b40da7971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59c55228aa4a48fc8da7b1a2040b18a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:59:35.531400 2546 tablet_service.cc:1467] Processing CreateTablet for tablet c534fd8160494e8f9c75852b40da7971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=59c55228aa4a48fc8da7b1a2040b18a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 02:59:35.532205 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c534fd8160494e8f9c75852b40da7971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.532202 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c534fd8160494e8f9c75852b40da7971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.532219 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c534fd8160494e8f9c75852b40da7971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.537969 2637 tablet_bootstrap.cc:492] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: Bootstrap starting.
I20260926 02:59:35.539283 2637 tablet_bootstrap.cc:654] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.539350 2635 tablet_bootstrap.cc:492] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: Bootstrap starting.
I20260926 02:59:35.539422 2636 tablet_bootstrap.cc:492] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: Bootstrap starting.
I20260926 02:59:35.540944 2635 tablet_bootstrap.cc:654] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.541383 2636 tablet_bootstrap.cc:654] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.543103 2637 tablet_bootstrap.cc:492] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: No bootstrap required, opened a new log
I20260926 02:59:35.543258 2637 ts_tablet_manager.cc:1397] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 02:59:35.543587 2635 tablet_bootstrap.cc:492] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: No bootstrap required, opened a new log
I20260926 02:59:35.543727 2635 ts_tablet_manager.cc:1397] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 02:59:35.543982 2637 raft_consensus.cc:348] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.544293 2636 tablet_bootstrap.cc:492] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: No bootstrap required, opened a new log
I20260926 02:59:35.544292 2637 raft_consensus.cc:374] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.544417 2636 ts_tablet_manager.cc:1397] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 02:59:35.544405 2637 raft_consensus.cc:729] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fc761fbe764257a2d0466ea73d3bdd, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.544764 2637 consensus_queue.cc:260] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.545162 2636 raft_consensus.cc:348] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.545389 2636 raft_consensus.cc:374] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.545406 2637 ts_tablet_manager.cc:1428] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:35.545485 2636 raft_consensus.cc:729] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0121773dadc34b1eb8f0cd3325f0ff04, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.545691 2636 consensus_queue.cc:260] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.545724 2642 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.545969 2642 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:35.546473 2636 ts_tablet_manager.cc:1428] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:35.546126 2641 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.546623 2641 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:35.546738 2635 raft_consensus.cc:348] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.546970 2635 raft_consensus.cc:374] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.547071 2635 raft_consensus.cc:729] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271ea220deeb492c8ad048b00492469f, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.547287 2635 consensus_queue.cc:260] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.547894 2635 ts_tablet_manager.cc:1428] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 02:59:35.547780 2643 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.548161 2643 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:35.610213 2642 raft_consensus.cc:482] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:35.610436 2642 raft_consensus.cc:504] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.611102 2642 leader_election.cc:288] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:35.611590 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534fd8160494e8f9c75852b40da7971" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" is_pre_election: true
I20260926 02:59:35.611596 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534fd8160494e8f9c75852b40da7971" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f" is_pre_election: true
I20260926 02:59:35.611860 2432 raft_consensus.cc:2454] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 0.
I20260926 02:59:35.611860 2571 raft_consensus.cc:2454] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 0.
I20260926 02:59:35.612257 2220 leader_election.cc:302] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271ea220deeb492c8ad048b00492469f, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:35.612531 2642 raft_consensus.cc:2781] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:35.612623 2642 raft_consensus.cc:482] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:35.612746 2642 raft_consensus.cc:3037] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.614686 2642 raft_consensus.cc:504] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.615178 2642 leader_election.cc:288] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 election: Requested vote from peers 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:35.615592 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534fd8160494e8f9c75852b40da7971" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f"
I20260926 02:59:35.615785 2571 raft_consensus.cc:3037] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.615723 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534fd8160494e8f9c75852b40da7971" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04"
I20260926 02:59:35.615993 2432 raft_consensus.cc:3037] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.617851 2571 raft_consensus.cc:2454] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 1.
I20260926 02:59:35.618232 2220 leader_election.cc:302] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271ea220deeb492c8ad048b00492469f, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:35.618263 2432 raft_consensus.cc:2454] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 1.
I20260926 02:59:35.618484 2642 raft_consensus.cc:2781] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:35.618676 2642 raft_consensus.cc:686] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 LEADER]: Becoming Leader. State: Replica: 39fc761fbe764257a2d0466ea73d3bdd, State: Running, Role: LEADER
I20260926 02:59:35.618954 2642 consensus_queue.cc:237] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.620739 2646 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: TxnStatusTablet state changed. Reason: New leader 39fc761fbe764257a2d0466ea73d3bdd. Latest consensus state: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.621188 2646 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:59:35.622043 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 02:59:35.622336 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 02:59:35.622079 2130 catalog_manager.cc:5495] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1). New cstate: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } health_report { overall_health: UNKNOWN } } }
I20260926 02:59:35.644194 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.637674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.97.1:37215)
0926 02:59:35.637844 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.637851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.638173 (+ 322us) server_negotiation.cc:407] Connection header received
0926 02:59:35.638493 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.638497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.638601 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.638756 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:35.639110 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.639966 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.641552 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.642122 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.642263 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.642842 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.643452 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.643619 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.643736 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.643817 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 02:59:35.644210 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.637664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37215 (local address 127.0.0.1:43666)
0926 02:59:35.638066 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.638085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.638274 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.638773 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.638778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.638791 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:35.638974 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.638983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.640111 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.640117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.641400 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.641413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.642285 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.642292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.642537 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.642592 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.643586 (+ 994us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.643606 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.643708 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.643864 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260926 02:59:35.652757 2432 raft_consensus.cc:1261] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Refusing update from remote peer 39fc761fbe764257a2d0466ea73d3bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:35.652760 2571 raft_consensus.cc:1261] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Refusing update from remote peer 39fc761fbe764257a2d0466ea73d3bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:35.653800 2642 consensus_queue.cc:1035] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:35.654404 2642 consensus_queue.cc:1035] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:35.670326 2643 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: TxnStatusTablet state changed. Reason: New leader 39fc761fbe764257a2d0466ea73d3bdd. Latest consensus state: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.670639 2643 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:35.675354 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333443893862965248 to 7333443893737414656. Current Snapshot: MvccSnapshot[applied={T|T < 7333443893862965248}]
I20260926 02:59:35.676270 2646 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.676545 2646 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:59:35.679216 2641 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: TxnStatusTablet state changed. Reason: New leader 39fc761fbe764257a2d0466ea73d3bdd. Latest consensus state: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.679558 2641 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:35.680830 2642 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.681182 2642 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 39fc761fbe764257a2d0466ea73d3bdd: This TxnStatusTablet replica's current role is: LEADER
I20260926 02:59:35.685648 2641 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.685889 2641 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 0121773dadc34b1eb8f0cd3325f0ff04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:35.690393 2643 tablet_replica.cc:439] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39fc761fbe764257a2d0466ea73d3bdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } } }
I20260926 02:59:35.690645 2643 tablet_replica.cc:442] T c534fd8160494e8f9c75852b40da7971 P 271ea220deeb492c8ad048b00492469f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 02:59:35.698760 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.693178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.0.97.1:37215)
0926 02:59:35.693373 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.693385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.693584 (+ 199us) server_negotiation.cc:407] Connection header received
0926 02:59:35.693761 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.693766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.693853 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.693952 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:35.694268 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.695430 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.696758 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.697427 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.697547 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.697924 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.698073 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.698172 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.698355 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.698428 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:59:35.715613 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.709651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42597 (local address 127.0.97.1:37215)
0926 02:59:35.709819 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.709825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.710366 (+ 541us) server_negotiation.cc:407] Connection header received
0926 02:59:35.710527 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.710531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.710595 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.710658 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:35.711218 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.711953 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.713983 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.714408 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.714500 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.714996 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.715095 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.715154 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.715306 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.715379 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":52}
I20260926 02:59:35.715945 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.709664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37215 (local address 127.0.97.1:42597)
0926 02:59:35.710263 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.710286 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.710414 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.710697 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.710702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.710713 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:35.711075 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.711085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.712086 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.712091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.713865 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.713879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.714574 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.714581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.714755 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.714817 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.715189 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.715201 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.715286 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.715454 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":20,"thread_start_us":311,"threads_started":1}
I20260926 02:59:35.725298 2432 raft_consensus.cc:1261] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Refusing update from remote peer 39fc761fbe764257a2d0466ea73d3bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:35.725338 2571 raft_consensus.cc:1261] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Refusing update from remote peer 39fc761fbe764257a2d0466ea73d3bdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:35.725831 2646 consensus_queue.cc:1035] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:35.726128 2642 consensus_queue.cc:1035] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:35.735956 2663 mvcc.cc:205] Tried to move back new op lower bound from 7333443894166786048 to 7333443893133832192. Current Snapshot: MvccSnapshot[applied={T|T < 7333443894166786048}]
I20260926 02:59:35.737496 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333443894166786048 to 7333443893133832192. Current Snapshot: MvccSnapshot[applied={T|T < 7333443894166786048}]
I20260926 02:59:35.805576 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:35.805836 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:35.809175 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Sending DeleteTablet for 3 replicas of tablet 43f20141c8fc4bf68d263eefd9dd3d78
I20260926 02:59:35.810146 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f20141c8fc4bf68d263eefd9dd3d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46968
I20260926 02:59:35.810552 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f20141c8fc4bf68d263eefd9dd3d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56514
I20260926 02:59:35.811350 2267 tablet_service.cc:1514] Processing DeleteTablet for tablet 43f20141c8fc4bf68d263eefd9dd3d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43652
I20260926 02:59:35.811694 2681 tablet_replica.cc:330] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: stopping tablet replica
I20260926 02:59:35.811795 2682 tablet_replica.cc:330] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: stopping tablet replica
I20260926 02:59:35.812225 2682 raft_consensus.cc:2229] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:35.812340 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:59:35.812567 2682 raft_consensus.cc:2258] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:35.812582 2681 raft_consensus.cc:2229] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:35.813201 2681 raft_consensus.cc:2258] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 02:59:35.812963 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:35.813472 2683 tablet_replica.cc:330] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: stopping tablet replica
I20260926 02:59:35.813828 2683 raft_consensus.cc:2229] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:35.814432 2683 raft_consensus.cc:2258] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:35.815310 2681 ts_tablet_manager.cc:1905] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:35.816588 2682 ts_tablet_manager.cc:1905] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:35.820497 2546 tablet_service.cc:1467] Processing CreateTablet for tablet cc59de86cefb46979110b33e35e90ca5 (DEFAULT_TABLE table=test-table [id=9146adb0a479467fa4b30010f0ed5c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:35.820684 2407 tablet_service.cc:1467] Processing CreateTablet for tablet cc59de86cefb46979110b33e35e90ca5 (DEFAULT_TABLE table=test-table [id=9146adb0a479467fa4b30010f0ed5c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:35.820896 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc59de86cefb46979110b33e35e90ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.821094 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc59de86cefb46979110b33e35e90ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.823906 2682 ts_tablet_manager.cc:1918] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:35.824038 2682 log.cc:1196] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/43f20141c8fc4bf68d263eefd9dd3d78
I20260926 02:59:35.824564 2682 ts_tablet_manager.cc:1939] T 43f20141c8fc4bf68d263eefd9dd3d78 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting consensus metadata
I20260926 02:59:35.825572 2267 tablet_service.cc:1467] Processing CreateTablet for tablet cc59de86cefb46979110b33e35e90ca5 (DEFAULT_TABLE table=test-table [id=9146adb0a479467fa4b30010f0ed5c3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:35.826046 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc59de86cefb46979110b33e35e90ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:35.827960 2636 tablet_bootstrap.cc:492] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: Bootstrap starting.
I20260926 02:59:35.828564 2635 tablet_bootstrap.cc:492] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: Bootstrap starting.
I20260926 02:59:35.829602 2636 tablet_bootstrap.cc:654] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.830178 2635 tablet_bootstrap.cc:654] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.832656 2637 tablet_bootstrap.cc:492] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: Bootstrap starting.
I20260926 02:59:35.834354 2637 tablet_bootstrap.cc:654] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:35.835831 2683 ts_tablet_manager.cc:1905] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:35.839810 2118 catalog_manager.cc:4841] TS 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561): tablet 43f20141c8fc4bf68d263eefd9dd3d78 (table test-table [id=47d0d483c7cb4d9db513d34cbd7fd9cd]) successfully deleted
I20260926 02:59:35.847493 2681 ts_tablet_manager.cc:1918] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:35.847816 2681 log.cc:1196] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/43f20141c8fc4bf68d263eefd9dd3d78
I20260926 02:59:35.848685 2635 tablet_bootstrap.cc:492] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: No bootstrap required, opened a new log
I20260926 02:59:35.850803 2635 ts_tablet_manager.cc:1397] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.001s
I20260926 02:59:35.851637 2635 raft_consensus.cc:348] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.851964 2635 raft_consensus.cc:374] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.852065 2635 raft_consensus.cc:729] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271ea220deeb492c8ad048b00492469f, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.852401 2635 consensus_queue.cc:260] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.852559 2683 ts_tablet_manager.cc:1918] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:35.852711 2683 log.cc:1196] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/43f20141c8fc4bf68d263eefd9dd3d78
I20260926 02:59:35.853312 2683 ts_tablet_manager.cc:1939] T 43f20141c8fc4bf68d263eefd9dd3d78 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting consensus metadata
I20260926 02:59:35.854902 2118 catalog_manager.cc:4841] TS 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215): tablet 43f20141c8fc4bf68d263eefd9dd3d78 (table test-table [id=47d0d483c7cb4d9db513d34cbd7fd9cd]) successfully deleted
I20260926 02:59:35.857069 2635 ts_tablet_manager.cc:1428] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 02:59:35.859619 2681 ts_tablet_manager.cc:1939] T 43f20141c8fc4bf68d263eefd9dd3d78 P 271ea220deeb492c8ad048b00492469f: Deleting consensus metadata
I20260926 02:59:35.861811 2117 catalog_manager.cc:4841] TS 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453): tablet 43f20141c8fc4bf68d263eefd9dd3d78 (table test-table [id=47d0d483c7cb4d9db513d34cbd7fd9cd]) successfully deleted
I20260926 02:59:35.863591 2636 tablet_bootstrap.cc:492] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: No bootstrap required, opened a new log
I20260926 02:59:35.863767 2636 ts_tablet_manager.cc:1397] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.007s
I20260926 02:59:35.864460 2636 raft_consensus.cc:348] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.864689 2637 tablet_bootstrap.cc:492] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: No bootstrap required, opened a new log
I20260926 02:59:35.864686 2636 raft_consensus.cc:374] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.864764 2636 raft_consensus.cc:729] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0121773dadc34b1eb8f0cd3325f0ff04, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.864847 2637 ts_tablet_manager.cc:1397] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.016s
I20260926 02:59:35.865025 2636 consensus_queue.cc:260] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.865594 2636 ts_tablet_manager.cc:1428] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 02:59:35.865550 2637 raft_consensus.cc:348] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.865787 2637 raft_consensus.cc:374] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:35.865860 2637 raft_consensus.cc:729] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fc761fbe764257a2d0466ea73d3bdd, State: Initialized, Role: FOLLOWER
I20260926 02:59:35.866118 2637 consensus_queue.cc:260] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.866850 2637 ts_tablet_manager.cc:1428] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:35.906105 2643 raft_consensus.cc:482] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:35.906340 2643 raft_consensus.cc:504] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.908154 2643 leader_election.cc:288] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215), 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561)
I20260926 02:59:35.915983 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.909435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58447 (local address 127.0.97.2:45561)
0926 02:59:35.909603 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.909610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.910185 (+ 575us) server_negotiation.cc:407] Connection header received
0926 02:59:35.910359 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.910367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.910461 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.910560 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:35.911017 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.912078 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.913891 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.915001 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.915130 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.915231 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.915540 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.915675 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:59:35.915995 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.909207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45561 (local address 127.0.97.3:58447)
0926 02:59:35.910065 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.910085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.910221 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.910579 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.910582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.910603 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:35.910883 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.910890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.912268 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.912277 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.913745 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.913756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.915190 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.915197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.915356 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.915388 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.915482 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.915602 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":1,"thread_start_us":567,"threads_started":1}
I20260926 02:59:35.916713 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.908510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55825 (local address 127.0.97.1:37215)
0926 02:59:35.908730 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.908736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.909154 (+ 418us) server_negotiation.cc:407] Connection header received
0926 02:59:35.909481 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.909490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.909607 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.909691 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:35.910430 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.911586 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.914145 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.915436 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.915588 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.915702 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.916249 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.916360 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:59:35.916849 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc59de86cefb46979110b33e35e90ca5" candidate_uuid: "271ea220deeb492c8ad048b00492469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" is_pre_election: true
I20260926 02:59:35.917116 2432 raft_consensus.cc:2454] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271ea220deeb492c8ad048b00492469f in term 0.
I20260926 02:59:35.917419 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.908435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37215 (local address 127.0.97.3:55825)
0926 02:59:35.909053 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.909079 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.909262 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.909936 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.909942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.909971 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:35.910248 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.910257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.911771 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.911776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.913996 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.914010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.915857 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.915864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.916049 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.916099 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.916887 (+ 788us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.917053 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":24,"thread_start_us":344,"threads_started":1}
I20260926 02:59:35.917677 2499 leader_election.cc:302] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0121773dadc34b1eb8f0cd3325f0ff04, 271ea220deeb492c8ad048b00492469f; no voters:
I20260926 02:59:35.918047 2643 raft_consensus.cc:2781] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:35.918183 2643 raft_consensus.cc:482] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:35.918283 2643 raft_consensus.cc:3037] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.918776 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc59de86cefb46979110b33e35e90ca5" candidate_uuid: "271ea220deeb492c8ad048b00492469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fc761fbe764257a2d0466ea73d3bdd" is_pre_election: true
I20260926 02:59:35.919179 2288 raft_consensus.cc:2454] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 271ea220deeb492c8ad048b00492469f in term 0.
I20260926 02:59:35.920663 2643 raft_consensus.cc:504] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.921252 2643 leader_election.cc:288] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [CANDIDATE]: Term 1 election: Requested vote from peers 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215), 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561)
I20260926 02:59:35.921729 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc59de86cefb46979110b33e35e90ca5" candidate_uuid: "271ea220deeb492c8ad048b00492469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fc761fbe764257a2d0466ea73d3bdd"
I20260926 02:59:35.921766 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc59de86cefb46979110b33e35e90ca5" candidate_uuid: "271ea220deeb492c8ad048b00492469f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04"
I20260926 02:59:35.921909 2288 raft_consensus.cc:3037] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.921947 2432 raft_consensus.cc:3037] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:35.924043 2288 raft_consensus.cc:2454] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271ea220deeb492c8ad048b00492469f in term 1.
I20260926 02:59:35.924422 2499 leader_election.cc:302] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271ea220deeb492c8ad048b00492469f, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:35.924688 2643 raft_consensus.cc:2781] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:35.924672 2432 raft_consensus.cc:2454] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 271ea220deeb492c8ad048b00492469f in term 1.
I20260926 02:59:35.930086 2643 raft_consensus.cc:686] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 1 LEADER]: Becoming Leader. State: Replica: 271ea220deeb492c8ad048b00492469f, State: Running, Role: LEADER
I20260926 02:59:35.930500 2643 consensus_queue.cc:237] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:35.933709 2132 catalog_manager.cc:5495] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f reported cstate change: term changed from 0 to 1, leader changed from <none> to 271ea220deeb492c8ad048b00492469f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "271ea220deeb492c8ad048b00492469f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } health_report { overall_health: HEALTHY } } }
I20260926 02:59:35.974663 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.969135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.0.97.3:45453)
0926 02:59:35.969623 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.969635 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.969648 (+ 13us) server_negotiation.cc:407] Connection header received
0926 02:59:35.969713 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.969716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.969791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.969959 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:35.970308 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.971131 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.972451 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.972845 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.972981 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.973426 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.974044 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.974202 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.974255 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.974326 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":219,"threads_started":1}
I20260926 02:59:35.990716 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.985236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57265 (local address 127.0.97.1:37215)
0926 02:59:35.985408 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:35.985413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:35.985951 (+ 538us) server_negotiation.cc:407] Connection header received
0926 02:59:35.986143 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:35.986148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:35.986216 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:35.986302 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:35.986786 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.987517 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.988859 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.989365 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.989503 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.989895 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.990012 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.990109 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:35.990252 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 02:59:35.990319 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 02:59:35.990758 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:35.985298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37215 (local address 127.0.97.3:57265)
0926 02:59:35.985857 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:35.985871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:35.986018 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:35.986331 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:35.986335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:35.986348 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:35.986647 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.986654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.987667 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.987675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.988717 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:35.988727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:35.989529 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:35.989535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:35.989669 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:35.989713 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:35.990110 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:35.990122 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:59:35.990225 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:59:35.990361 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":329,"threads_started":1}
I20260926 02:59:35.999806 2432 raft_consensus.cc:1261] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Refusing update from remote peer 271ea220deeb492c8ad048b00492469f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:36.000218 2288 raft_consensus.cc:1261] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Refusing update from remote peer 271ea220deeb492c8ad048b00492469f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:36.001012 2689 consensus_queue.cc:1035] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:36.001478 2643 consensus_queue.cc:1035] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [LEADER]: Connected to new peer: Peer: permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:36.027987 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:36.022298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36489 (local address 127.0.97.3:45453)
0926 02:59:36.022445 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:36.022452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:36.022469 (+ 17us) server_negotiation.cc:407] Connection header received
0926 02:59:36.022523 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:36.022528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:36.022603 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:36.022694 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:36.023080 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:36.024164 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:36.026237 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:36.026874 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:36.026998 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:36.027338 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:36.027426 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:36.027520 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:36.027624 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 02:59:36.027704 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 02:59:36.028039 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:36.022098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45453 (local address 127.0.97.1:36489)
0926 02:59:36.022265 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:36.022285 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:36.022397 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:36.022716 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:36.022721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:36.022737 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:36.022932 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:36.022943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:36.024317 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:36.024326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:36.026108 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:36.026120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:36.027016 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:36.027020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:36.027149 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:36.027182 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:36.027519 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:36.027531 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 02:59:36.027607 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 02:59:36.027754 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 02:59:36.029081 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:59:37.642534 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:37.642800 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:37.646937 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Sending DeleteTablet for 3 replicas of tablet cc59de86cefb46979110b33e35e90ca5
I20260926 02:59:37.647723 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet cc59de86cefb46979110b33e35e90ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43652
I20260926 02:59:37.647744 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet cc59de86cefb46979110b33e35e90ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56514
I20260926 02:59:37.647897 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet cc59de86cefb46979110b33e35e90ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46968
I20260926 02:59:37.658937 2724 tablet_replica.cc:330] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: stopping tablet replica
I20260926 02:59:37.659114 2722 tablet_replica.cc:330] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: stopping tablet replica
I20260926 02:59:37.659152 2724 raft_consensus.cc:2229] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:37.659292 2722 raft_consensus.cc:2229] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:37.659524 2722 raft_consensus.cc:2258] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:37.659705 2724 raft_consensus.cc:2258] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:37.660431 2722 ts_tablet_manager.cc:1905] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:37.661058 2724 ts_tablet_manager.cc:1905] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:37.666360 2723 tablet_replica.cc:330] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: stopping tablet replica
I20260926 02:59:37.666549 2723 raft_consensus.cc:2229] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:37.666781 2723 raft_consensus.cc:2258] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:37.667317 2722 ts_tablet_manager.cc:1918] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:59:37.667452 2722 log.cc:1196] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/cc59de86cefb46979110b33e35e90ca5
I20260926 02:59:37.667665 2723 ts_tablet_manager.cc:1905] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:37.667889 2722 ts_tablet_manager.cc:1939] T cc59de86cefb46979110b33e35e90ca5 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting consensus metadata
I20260926 02:59:37.668938 2118 catalog_manager.cc:4841] TS 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215): tablet cc59de86cefb46979110b33e35e90ca5 (table test-table [id=9146adb0a479467fa4b30010f0ed5c3b]) successfully deleted
I20260926 02:59:37.669206 2724 ts_tablet_manager.cc:1918] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:59:37.669389 2724 log.cc:1196] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/cc59de86cefb46979110b33e35e90ca5
I20260926 02:59:37.669912 2724 ts_tablet_manager.cc:1939] T cc59de86cefb46979110b33e35e90ca5 P 271ea220deeb492c8ad048b00492469f: Deleting consensus metadata
I20260926 02:59:37.671068 2117 catalog_manager.cc:4841] TS 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453): tablet cc59de86cefb46979110b33e35e90ca5 (table test-table [id=9146adb0a479467fa4b30010f0ed5c3b]) successfully deleted
I20260926 02:59:37.674405 2723 ts_tablet_manager.cc:1918] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:59:37.674542 2723 log.cc:1196] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/cc59de86cefb46979110b33e35e90ca5
I20260926 02:59:37.674958 2723 ts_tablet_manager.cc:1939] T cc59de86cefb46979110b33e35e90ca5 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting consensus metadata
I20260926 02:59:37.676052 2118 catalog_manager.cc:4841] TS 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561): tablet cc59de86cefb46979110b33e35e90ca5 (table test-table [id=9146adb0a479467fa4b30010f0ed5c3b]) successfully deleted
Sep 26 02:59:37 dist-test-slave-fccw krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 02:59:37 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:37.708719 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.691596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.0.97.62:43715)
0926 02:59:37.692087 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:37.692092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:37.692208 (+ 116us) server_negotiation.cc:407] Connection header received
0926 02:59:37.692367 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:37.692371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:37.692429 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:37.692520 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:37.693797 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.694533 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.695662 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.696028 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.696124 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.703928 (+ 7804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:37.703933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:37.703936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:37.703964 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:37.706791 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:37.707346 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:37.707349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:37.707351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:37.707459 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:37.707733 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:37.707736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:37.707738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:37.707959 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:37.708112 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:37.708338 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 02:59:37.708476 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":249,"threads_started":1}
I20260926 02:59:37.711759 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:59:37.712345 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:37.718242 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 54034ce50a6442af97bf670dbd602aab (DEFAULT_TABLE table=test-table [id=0b9a38e885364d05abea759ee582724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:37.718293 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 54034ce50a6442af97bf670dbd602aab (DEFAULT_TABLE table=test-table [id=0b9a38e885364d05abea759ee582724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:37.718523 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 54034ce50a6442af97bf670dbd602aab (DEFAULT_TABLE table=test-table [id=0b9a38e885364d05abea759ee582724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:37.718662 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54034ce50a6442af97bf670dbd602aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:37.718658 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54034ce50a6442af97bf670dbd602aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:37.718938 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54034ce50a6442af97bf670dbd602aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:37.724789 2733 tablet_bootstrap.cc:492] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: Bootstrap starting.
I20260926 02:59:37.725425 2734 tablet_bootstrap.cc:492] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: Bootstrap starting.
I20260926 02:59:37.726585 2733 tablet_bootstrap.cc:654] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:37.727047 2734 tablet_bootstrap.cc:654] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:37.728500 2735 tablet_bootstrap.cc:492] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: Bootstrap starting.
I20260926 02:59:37.729844 2733 tablet_bootstrap.cc:492] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: No bootstrap required, opened a new log
I20260926 02:59:37.729970 2733 ts_tablet_manager.cc:1397] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 02:59:37.730674 2733 raft_consensus.cc:348] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.730890 2733 raft_consensus.cc:374] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:37.730985 2733 raft_consensus.cc:729] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271ea220deeb492c8ad048b00492469f, State: Initialized, Role: FOLLOWER
I20260926 02:59:37.731226 2733 consensus_queue.cc:260] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.732018 2734 tablet_bootstrap.cc:492] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: No bootstrap required, opened a new log
I20260926 02:59:37.732172 2734 ts_tablet_manager.cc:1397] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 02:59:37.732844 2734 raft_consensus.cc:348] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.733100 2734 raft_consensus.cc:374] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:37.733191 2734 raft_consensus.cc:729] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fc761fbe764257a2d0466ea73d3bdd, State: Initialized, Role: FOLLOWER
I20260926 02:59:37.733454 2734 consensus_queue.cc:260] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.734057 2734 ts_tablet_manager.cc:1428] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:37.734825 2735 tablet_bootstrap.cc:654] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:37.735770 2733 ts_tablet_manager.cc:1428] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260926 02:59:37.739063 2735 tablet_bootstrap.cc:492] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: No bootstrap required, opened a new log
I20260926 02:59:37.739193 2735 ts_tablet_manager.cc:1397] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20260926 02:59:37.739882 2735 raft_consensus.cc:348] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.740121 2735 raft_consensus.cc:374] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:37.740203 2735 raft_consensus.cc:729] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0121773dadc34b1eb8f0cd3325f0ff04, State: Initialized, Role: FOLLOWER
I20260926 02:59:37.740429 2735 consensus_queue.cc:260] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.741307 2735 ts_tablet_manager.cc:1428] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 02:59:37.777103 2739 raft_consensus.cc:482] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:37.777325 2739 raft_consensus.cc:504] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.778991 2739 leader_election.cc:288] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:37.796425 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.779220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34387 (local address 127.0.97.1:37215)
0926 02:59:37.789290 (+ 10070us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:37.789296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:37.789546 (+ 250us) server_negotiation.cc:407] Connection header received
0926 02:59:37.789850 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:37.789859 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:37.789946 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:37.790046 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:37.790697 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.791805 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.794270 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.795413 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.795550 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.795677 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:37.795950 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 02:59:37.796043 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9905,"thread_start_us":9778,"threads_started":1}
I20260926 02:59:37.796515 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.779195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45453 (local address 127.0.97.2:54191)
0926 02:59:37.790029 (+ 10834us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:37.790046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:37.790140 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:37.790751 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:37.790754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:37.790773 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:37.791049 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.791054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.792241 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.792263 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.794317 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.794325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.795788 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.795793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.795986 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.796013 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 02:59:37.796084 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 02:59:37.796186 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10703,"mutex_wait_us":23,"thread_start_us":10113,"threads_started":1}
I20260926 02:59:37.796550 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.779399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54191 (local address 127.0.97.3:45453)
0926 02:59:37.790461 (+ 11062us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:37.790466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:37.790486 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:59:37.790540 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:37.790543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:37.790618 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:37.790729 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:37.791202 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.792132 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.794442 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.795591 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.795708 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.795802 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:37.796126 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 02:59:37.796251 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10890,"thread_start_us":10757,"threads_started":1}
I20260926 02:59:37.796793 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.779172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37215 (local address 127.0.97.2:34387)
0926 02:59:37.789459 (+ 10287us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:37.789479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:37.789627 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:37.790266 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:37.790270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:37.790299 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:37.790527 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.790535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.791944 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.791947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.794119 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.794132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.795578 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.795582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.795776 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.795825 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 02:59:37.795936 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 02:59:37.796061 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10171,"mutex_wait_us":3,"thread_start_us":10554,"threads_started":1}
I20260926 02:59:37.796455 2678 raft_consensus.cc:482] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:37.797076 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f" is_pre_election: true
I20260926 02:59:37.797339 2570 raft_consensus.cc:2454] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in term 0.
I20260926 02:59:37.797487 2678 raft_consensus.cc:504] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.797755 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fc761fbe764257a2d0466ea73d3bdd" is_pre_election: true
I20260926 02:59:37.797816 2359 leader_election.cc:302] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0121773dadc34b1eb8f0cd3325f0ff04, 271ea220deeb492c8ad048b00492469f; no voters:
I20260926 02:59:37.797935 2288 raft_consensus.cc:2454] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in term 0.
I20260926 02:59:37.798199 2739 raft_consensus.cc:2781] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:37.798214 2678 leader_election.cc:288] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:37.798314 2739 raft_consensus.cc:482] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:37.798408 2739 raft_consensus.cc:3037] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:37.798856 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f" is_pre_election: true
I20260926 02:59:37.798915 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" is_pre_election: true
I20260926 02:59:37.799108 2570 raft_consensus.cc:2454] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39fc761fbe764257a2d0466ea73d3bdd in term 0.
I20260926 02:59:37.799465 2220 leader_election.cc:302] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 271ea220deeb492c8ad048b00492469f, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:37.799744 2678 raft_consensus.cc:2781] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:37.799865 2678 raft_consensus.cc:482] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:37.799953 2678 raft_consensus.cc:3037] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:37.801612 2739 raft_consensus.cc:504] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.801982 2432 raft_consensus.cc:2379] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39fc761fbe764257a2d0466ea73d3bdd in current term 1: Already voted for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in this term.
I20260926 02:59:37.802239 2739 leader_election.cc:288] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 election: Requested vote from peers 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:37.802764 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f"
I20260926 02:59:37.802789 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fc761fbe764257a2d0466ea73d3bdd"
I20260926 02:59:37.802965 2570 raft_consensus.cc:3037] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:37.802903 2678 raft_consensus.cc:504] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.803314 2288 raft_consensus.cc:2379] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0121773dadc34b1eb8f0cd3325f0ff04 in current term 1: Already voted for candidate 39fc761fbe764257a2d0466ea73d3bdd in this term.
I20260926 02:59:37.803530 2678 leader_election.cc:288] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 election: Requested vote from peers 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:37.804128 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0121773dadc34b1eb8f0cd3325f0ff04"
I20260926 02:59:37.804345 2432 raft_consensus.cc:2379] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39fc761fbe764257a2d0466ea73d3bdd in current term 1: Already voted for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in this term.
I20260926 02:59:37.805184 2570 raft_consensus.cc:2454] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in term 1.
I20260926 02:59:37.805373 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54034ce50a6442af97bf670dbd602aab" candidate_uuid: "39fc761fbe764257a2d0466ea73d3bdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f"
I20260926 02:59:37.805572 2359 leader_election.cc:302] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0121773dadc34b1eb8f0cd3325f0ff04, 271ea220deeb492c8ad048b00492469f; no voters: 39fc761fbe764257a2d0466ea73d3bdd
I20260926 02:59:37.805627 2571 raft_consensus.cc:2379] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39fc761fbe764257a2d0466ea73d3bdd in current term 1: Already voted for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in this term.
I20260926 02:59:37.805876 2739 raft_consensus.cc:2781] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:37.806051 2220 leader_election.cc:302] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39fc761fbe764257a2d0466ea73d3bdd; no voters: 0121773dadc34b1eb8f0cd3325f0ff04, 271ea220deeb492c8ad048b00492469f
I20260926 02:59:37.806355 2678 raft_consensus.cc:2726] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 02:59:37.806350 2739 raft_consensus.cc:686] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 LEADER]: Becoming Leader. State: Replica: 0121773dadc34b1eb8f0cd3325f0ff04, State: Running, Role: LEADER
I20260926 02:59:37.806725 2739 consensus_queue.cc:237] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.810109 2130 catalog_manager.cc:5495] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } health_report { overall_health: UNKNOWN } } }
I20260926 02:59:37.834681 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.829054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.0.97.2:45561)
0926 02:59:37.829585 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:37.829590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:37.829623 (+ 33us) server_negotiation.cc:407] Connection header received
0926 02:59:37.829691 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:37.829694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:37.829754 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:37.829914 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:37.830317 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.831091 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.832349 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.832747 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.832877 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.833384 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:37.833979 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:37.834165 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:37.834267 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 02:59:37.834337 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":266,"threads_started":1}
I20260926 02:59:37.848920 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.844019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52109 (local address 127.0.97.1:37215)
0926 02:59:37.844272 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:37.844278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:37.844586 (+ 308us) server_negotiation.cc:407] Connection header received
0926 02:59:37.844782 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:37.844787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:37.844845 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:37.844935 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:37.845413 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.846053 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.847345 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.847735 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.847868 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.848250 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:37.848347 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:37.848438 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:37.848567 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 02:59:37.848633 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 02:59:37.849061 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.843844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37215 (local address 127.0.97.2:52109)
0926 02:59:37.844496 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:37.844511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:37.844657 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:37.844946 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:37.844951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:37.844962 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:37.845282 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.845290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.846183 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.846189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.847193 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.847203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.847897 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.847903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.848041 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.848091 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:37.848439 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:37.848450 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 02:59:37.848543 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 02:59:37.848662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":395,"threads_started":1}
I20260926 02:59:37.856253 2571 raft_consensus.cc:1261] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Refusing update from remote peer 0121773dadc34b1eb8f0cd3325f0ff04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:37.856380 2288 raft_consensus.cc:1261] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Refusing update from remote peer 0121773dadc34b1eb8f0cd3325f0ff04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:37.856861 2744 consensus_queue.cc:1035] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:37.857306 2739 consensus_queue.cc:1035] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:37.882867 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.876520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34671 (local address 127.0.97.2:45561)
0926 02:59:37.876694 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:37.876701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:37.877291 (+ 590us) server_negotiation.cc:407] Connection header received
0926 02:59:37.877568 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:37.877573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:37.877655 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:37.877751 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:37.878577 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.879717 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.881285 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.881675 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.881812 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.882140 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:37.882275 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:37.882345 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:37.882497 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 02:59:37.882572 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 02:59:37.882982 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:37.876464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45561 (local address 127.0.97.1:34671)
0926 02:59:37.877190 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:37.877212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:37.877397 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:37.877817 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:37.877822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:37.877833 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 02:59:37.878186 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.878218 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.879884 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.879888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.881171 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:37.881182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:37.881848 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:37.881852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:37.881973 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:37.882006 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 02:59:37.882369 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 02:59:37.882379 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 02:59:37.882473 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 02:59:37.882576 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":362,"threads_started":1}
I20260926 02:59:37.911921 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:37.912150 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:37.915050 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Sending DeleteTablet for 3 replicas of tablet 54034ce50a6442af97bf670dbd602aab
I20260926 02:59:37.915818 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 54034ce50a6442af97bf670dbd602aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43652
I20260926 02:59:37.915838 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 54034ce50a6442af97bf670dbd602aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56514
I20260926 02:59:37.916113 2722 tablet_replica.cc:330] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: stopping tablet replica
I20260926 02:59:37.916307 2722 raft_consensus.cc:2229] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:37.916400 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 54034ce50a6442af97bf670dbd602aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46968
I20260926 02:59:37.916498 2722 raft_consensus.cc:2258] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:37.916678 2724 tablet_replica.cc:330] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: stopping tablet replica
I20260926 02:59:37.916777 2723 tablet_replica.cc:330] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: stopping tablet replica
I20260926 02:59:37.916857 2724 raft_consensus.cc:2229] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:37.916957 2723 raft_consensus.cc:2229] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:37.917057 2724 raft_consensus.cc:2258] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:37.917470 2723 raft_consensus.cc:2258] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:37.918097 2724 ts_tablet_manager.cc:1905] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:37.918824 2722 ts_tablet_manager.cc:1905] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:37.918933 2723 ts_tablet_manager.cc:1905] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:37.924396 2724 ts_tablet_manager.cc:1918] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:37.924543 2724 log.cc:1196] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/54034ce50a6442af97bf670dbd602aab
I20260926 02:59:37.924938 2724 ts_tablet_manager.cc:1939] T 54034ce50a6442af97bf670dbd602aab P 271ea220deeb492c8ad048b00492469f: Deleting consensus metadata
I20260926 02:59:37.925619 2722 ts_tablet_manager.cc:1918] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:37.925827 2722 log.cc:1196] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/54034ce50a6442af97bf670dbd602aab
I20260926 02:59:37.925714 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 02:59:37.926311 2722 ts_tablet_manager.cc:1939] T 54034ce50a6442af97bf670dbd602aab P 39fc761fbe764257a2d0466ea73d3bdd: Deleting consensus metadata
W20260926 02:59:37.926432 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:37.927253 2117 catalog_manager.cc:4841] TS 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453): tablet 54034ce50a6442af97bf670dbd602aab (table test-table [id=0b9a38e885364d05abea759ee582724c]) successfully deleted
I20260926 02:59:37.927285 2118 catalog_manager.cc:4841] TS 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215): tablet 54034ce50a6442af97bf670dbd602aab (table test-table [id=0b9a38e885364d05abea759ee582724c]) successfully deleted
I20260926 02:59:37.927995 2723 ts_tablet_manager.cc:1918] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 02:59:37.928153 2723 log.cc:1196] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/54034ce50a6442af97bf670dbd602aab
I20260926 02:59:37.928536 2723 ts_tablet_manager.cc:1939] T 54034ce50a6442af97bf670dbd602aab P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting consensus metadata
I20260926 02:59:37.930017 2118 catalog_manager.cc:4841] TS 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561): tablet 54034ce50a6442af97bf670dbd602aab (table test-table [id=0b9a38e885364d05abea759ee582724c]) successfully deleted
I20260926 02:59:37.933065 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4c748f28d64d1da183e9d05fd4c6a6 (DEFAULT_TABLE table=test-table [id=05cb287fcd2544ab9d1ce7efdb3c422b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:37.933279 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4c748f28d64d1da183e9d05fd4c6a6 (DEFAULT_TABLE table=test-table [id=05cb287fcd2544ab9d1ce7efdb3c422b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:37.933447 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4c748f28d64d1da183e9d05fd4c6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:37.933656 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4c748f28d64d1da183e9d05fd4c6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:37.936825 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4c748f28d64d1da183e9d05fd4c6a6 (DEFAULT_TABLE table=test-table [id=05cb287fcd2544ab9d1ce7efdb3c422b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:37.937285 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d4c748f28d64d1da183e9d05fd4c6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:37.939479 2734 tablet_bootstrap.cc:492] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: Bootstrap starting.
I20260926 02:59:37.940909 2733 tablet_bootstrap.cc:492] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: Bootstrap starting.
I20260926 02:59:37.941113 2734 tablet_bootstrap.cc:654] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:37.942854 2733 tablet_bootstrap.cc:654] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:37.944667 2735 tablet_bootstrap.cc:492] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: Bootstrap starting.
I20260926 02:59:37.946297 2735 tablet_bootstrap.cc:654] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:37.949666 2733 tablet_bootstrap.cc:492] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: No bootstrap required, opened a new log
I20260926 02:59:37.949827 2733 ts_tablet_manager.cc:1397] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 02:59:37.949986 2734 tablet_bootstrap.cc:492] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: No bootstrap required, opened a new log
I20260926 02:59:37.950138 2734 ts_tablet_manager.cc:1397] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 02:59:37.950534 2733 raft_consensus.cc:348] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.950817 2733 raft_consensus.cc:374] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:37.950898 2733 raft_consensus.cc:729] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271ea220deeb492c8ad048b00492469f, State: Initialized, Role: FOLLOWER
I20260926 02:59:37.950845 2734 raft_consensus.cc:348] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.951072 2734 raft_consensus.cc:374] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:37.951165 2734 raft_consensus.cc:729] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fc761fbe764257a2d0466ea73d3bdd, State: Initialized, Role: FOLLOWER
I20260926 02:59:37.951135 2733 consensus_queue.cc:260] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.951378 2734 consensus_queue.cc:260] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.951689 2733 ts_tablet_manager.cc:1428] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:37.952934 2734 ts_tablet_manager.cc:1428] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 02:59:37.957757 2735 tablet_bootstrap.cc:492] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: No bootstrap required, opened a new log
I20260926 02:59:37.957933 2735 ts_tablet_manager.cc:1397] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260926 02:59:37.958702 2735 raft_consensus.cc:348] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.958950 2735 raft_consensus.cc:374] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:37.959043 2735 raft_consensus.cc:729] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0121773dadc34b1eb8f0cd3325f0ff04, State: Initialized, Role: FOLLOWER
I20260926 02:59:37.959259 2735 consensus_queue.cc:260] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:37.959807 2735 ts_tablet_manager.cc:1428] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 02:59:38.207543 2739 raft_consensus.cc:482] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:38.207818 2739 raft_consensus.cc:504] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:38.208519 2739 leader_election.cc:288] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:38.209059 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c748f28d64d1da183e9d05fd4c6a6" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fc761fbe764257a2d0466ea73d3bdd" is_pre_election: true
I20260926 02:59:38.209175 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c748f28d64d1da183e9d05fd4c6a6" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f" is_pre_election: true
I20260926 02:59:38.209311 2288 raft_consensus.cc:2454] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in term 0.
I20260926 02:59:38.209404 2571 raft_consensus.cc:2454] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in term 0.
I20260926 02:59:38.209807 2360 leader_election.cc:302] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0121773dadc34b1eb8f0cd3325f0ff04, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:38.210430 2739 raft_consensus.cc:2781] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:38.210565 2739 raft_consensus.cc:482] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:38.210685 2739 raft_consensus.cc:3037] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:38.213901 2739 raft_consensus.cc:504] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:38.214542 2739 leader_election.cc:288] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 election: Requested vote from peers 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215), 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453)
I20260926 02:59:38.214833 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c748f28d64d1da183e9d05fd4c6a6" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fc761fbe764257a2d0466ea73d3bdd"
I20260926 02:59:38.215049 2288 raft_consensus.cc:3037] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:38.215128 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d4c748f28d64d1da183e9d05fd4c6a6" candidate_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "271ea220deeb492c8ad048b00492469f"
I20260926 02:59:38.215358 2571 raft_consensus.cc:3037] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:38.217839 2288 raft_consensus.cc:2454] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in term 1.
I20260926 02:59:38.218221 2571 raft_consensus.cc:2454] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0121773dadc34b1eb8f0cd3325f0ff04 in term 1.
I20260926 02:59:38.218250 2360 leader_election.cc:302] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0121773dadc34b1eb8f0cd3325f0ff04, 39fc761fbe764257a2d0466ea73d3bdd; no voters:
I20260926 02:59:38.218525 2739 raft_consensus.cc:2781] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:38.218657 2739 raft_consensus.cc:686] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 LEADER]: Becoming Leader. State: Replica: 0121773dadc34b1eb8f0cd3325f0ff04, State: Running, Role: LEADER
I20260926 02:59:38.218997 2739 consensus_queue.cc:237] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } }
I20260926 02:59:38.238533 2131 catalog_manager.cc:5495] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0121773dadc34b1eb8f0cd3325f0ff04" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 } health_report { overall_health: UNKNOWN } } }
I20260926 02:59:38.321245 2288 raft_consensus.cc:1261] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Refusing update from remote peer 0121773dadc34b1eb8f0cd3325f0ff04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:38.321739 2571 raft_consensus.cc:1261] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Refusing update from remote peer 0121773dadc34b1eb8f0cd3325f0ff04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:38.321725 2739 consensus_queue.cc:1035] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39fc761fbe764257a2d0466ea73d3bdd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:38.322263 2744 consensus_queue.cc:1035] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "271ea220deeb492c8ad048b00492469f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 02:59:38.341508 2703 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 02:59:39.936080 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:39.936312 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:39.940093 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d54a736388d47eaae8103af55dde138: Sending DeleteTablet for 3 replicas of tablet 2d4c748f28d64d1da183e9d05fd4c6a6
I20260926 02:59:39.940900 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4c748f28d64d1da183e9d05fd4c6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56514
I20260926 02:59:39.941295 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4c748f28d64d1da183e9d05fd4c6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43652
I20260926 02:59:39.941722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2203
I20260926 02:59:39.941725 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d4c748f28d64d1da183e9d05fd4c6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46968
I20260926 02:59:39.942005 2786 tablet_replica.cc:330] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: stopping tablet replica
I20260926 02:59:39.942268 2786 raft_consensus.cc:2229] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:39.942854 2786 raft_consensus.cc:2258] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:39.944645 2786 ts_tablet_manager.cc:1905] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:39.951124 2788 tablet_replica.cc:330] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: stopping tablet replica
I20260926 02:59:39.951382 2788 raft_consensus.cc:2229] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:39.951632 2788 raft_consensus.cc:2258] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:39.952698 2788 ts_tablet_manager.cc:1905] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:39.953070 2786 ts_tablet_manager.cc:1918] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:59:39.953253 2786 log.cc:1196] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/2d4c748f28d64d1da183e9d05fd4c6a6
I20260926 02:59:39.953768 2786 ts_tablet_manager.cc:1939] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 0121773dadc34b1eb8f0cd3325f0ff04: Deleting consensus metadata
I20260926 02:59:39.954319 2787 tablet_replica.cc:330] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: stopping tablet replica
I20260926 02:59:39.954505 2787 raft_consensus.cc:2229] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:39.954738 2787 raft_consensus.cc:2258] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:39.955515 2787 ts_tablet_manager.cc:1905] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:39.956251 2118 catalog_manager.cc:4841] TS 0121773dadc34b1eb8f0cd3325f0ff04 (127.0.97.2:45561): tablet 2d4c748f28d64d1da183e9d05fd4c6a6 (table test-table [id=05cb287fcd2544ab9d1ce7efdb3c422b]) successfully deleted
I20260926 02:59:39.958913 2788 ts_tablet_manager.cc:1918] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:59:39.959052 2788 log.cc:1196] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/2d4c748f28d64d1da183e9d05fd4c6a6
I20260926 02:59:39.959539 2788 ts_tablet_manager.cc:1939] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 271ea220deeb492c8ad048b00492469f: Deleting consensus metadata
I20260926 02:59:39.960649 2117 catalog_manager.cc:4841] TS 271ea220deeb492c8ad048b00492469f (127.0.97.3:45453): tablet 2d4c748f28d64d1da183e9d05fd4c6a6 (table test-table [id=05cb287fcd2544ab9d1ce7efdb3c422b]) successfully deleted
I20260926 02:59:39.960881 2787 ts_tablet_manager.cc:1918] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 02:59:39.961045 2787 log.cc:1196] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/2d4c748f28d64d1da183e9d05fd4c6a6
I20260926 02:59:39.961418 2787 ts_tablet_manager.cc:1939] T 2d4c748f28d64d1da183e9d05fd4c6a6 P 39fc761fbe764257a2d0466ea73d3bdd: Deleting consensus metadata
I20260926 02:59:39.962316 2118 catalog_manager.cc:4841] TS 39fc761fbe764257a2d0466ea73d3bdd (127.0.97.1:37215): tablet 2d4c748f28d64d1da183e9d05fd4c6a6 (table test-table [id=05cb287fcd2544ab9d1ce7efdb3c422b]) successfully deleted
Sep 26 02:59:39 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:39.967962 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:39.944391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.0.97.1:37215)
0926 02:59:39.951306 (+ 6915us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:39.951313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:39.951888 (+ 575us) server_negotiation.cc:407] Connection header received
0926 02:59:39.951951 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:39.951956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:39.952025 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:39.952128 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:39.953959 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:39.955108 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:39.957241 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:39.957879 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:39.957998 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:39.963277 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:39.963291 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:39.963306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:39.963345 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:39.965868 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:39.966429 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:39.966436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:39.966438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:39.966553 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:39.966849 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:39.966852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:39.966853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:39.967183 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:39.967388 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:39.967577 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 02:59:39.967729 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6781,"thread_start_us":6657,"threads_started":1}
W20260926 02:59:39.968614 2333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43680
W20260926 02:59:39.969229 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:39.969401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 2203
I20260926 02:59:39.994645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2342
Sep 26 02:59:40 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 02:59:40.008644 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:39.995318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.97.2:45561)
0926 02:59:39.995805 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:39.995809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:39.995824 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:59:39.995872 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:39.995876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:39.995933 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:39.996047 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:39.997293 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:39.998082 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:39.999246 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:39.999672 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:39.999775 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:40.003879 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:40.003891 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:40.003905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:40.003939 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:40.006490 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:40.007033 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:40.007039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:40.007041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:40.007145 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:40.007439 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:40.007441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:40.007443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:40.007774 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:40.007983 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:40.008229 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 02:59:40.008375 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":281,"threads_started":1}
W20260926 02:59:40.009289 2472 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56536
W20260926 02:59:40.009840 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:40.009986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 2342
I20260926 02:59:40.030867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2481
Sep 26 02:59:40 dist-test-slave-fccw krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 02:59:40.045403 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:40.031622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.0.97.3:45453)
0926 02:59:40.032114 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:40.032118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:40.032133 (+ 15us) server_negotiation.cc:407] Connection header received
0926 02:59:40.032182 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:40.032186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:40.032238 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:40.032323 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:40.033550 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:40.034256 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:40.035363 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:40.035746 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:40.035830 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:40.040079 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:40.040092 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:40.040108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:40.040136 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:40.042828 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:40.043458 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:40.043466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:40.043468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:40.043587 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:40.043929 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:40.043936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:40.043939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:40.044269 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:40.044587 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:40.044843 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 02:59:40.045083 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
W20260926 02:59:40.046106 2611 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47000
W20260926 02:59:40.046657 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2481: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:40.046829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 2481
I20260926 02:59:40.068301 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2100
I20260926 02:59:40.069406 2172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:40.191576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 2100
2026-09-26T02:59:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8184 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:40 dist-test-slave-fccw krb5kdc[2796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:40 dist-test-slave-fccw krb5kdc[2796](info): set up 2 sockets
Sep 26 02:59:40 dist-test-slave-fccw krb5kdc[2796](info): commencing operation
krb5kdc: starting...
W20260926 02:59:41.249217 388 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:41 dist-test-slave-fccw krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:41.438079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:41.750013 2809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:41.750394 2809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:41.750553 2809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:41.757508 2809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:41.757642 2809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:41.757740 2809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:41.757825 2809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:41.757890 2809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:41.767107 2809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:41.768313 2809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:41.769605 2809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:41.781052 2815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:41.781015 2814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:41.781080 2817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:41.781679 2809 server_base.cc:1034] running on GCE node
I20260926 02:59:41.782387 2809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:41.783675 2809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:41.784889 2809 hybrid_clock.cc:648] HybridClock initialized: now 1790391581784855 us; error 46 us; skew 500 ppm
I20260926 02:59:41.785274 2809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:41 dist-test-slave-fccw krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:41.788137 2809 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:41.789613 2809 webserver.cc:460] Webserver started at http://127.0.97.62:41563/ using document root <none> and password file <none>
I20260926 02:59:41.790370 2809 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:41.790513 2809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:41.790871 2809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:41.793706 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "569e0cb0ee9f4006a7c7a9ee41709de0"
format_stamp: "Formatted at 2026-09-26 02:59:41 on dist-test-slave-fccw"
server_key: "000b7e0052a81f9ec9ac90b73a4da968"
server_key_iv: "f46740a87c19ef94572cdf7101a7c2b5"
server_key_version: "encryptionkey@0"
I20260926 02:59:41.794472 2809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "569e0cb0ee9f4006a7c7a9ee41709de0"
format_stamp: "Formatted at 2026-09-26 02:59:41 on dist-test-slave-fccw"
server_key: "000b7e0052a81f9ec9ac90b73a4da968"
server_key_iv: "f46740a87c19ef94572cdf7101a7c2b5"
server_key_version: "encryptionkey@0"
I20260926 02:59:41.799175 2809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:59:41.802407 2824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:41.803570 2809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 02:59:41.803740 2809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "569e0cb0ee9f4006a7c7a9ee41709de0"
format_stamp: "Formatted at 2026-09-26 02:59:41 on dist-test-slave-fccw"
server_key: "000b7e0052a81f9ec9ac90b73a4da968"
server_key_iv: "f46740a87c19ef94572cdf7101a7c2b5"
server_key_version: "encryptionkey@0"
I20260926 02:59:41.803921 2809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:41.821406 2809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:41.824987 2809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:41.825274 2809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:41.842938 2809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42829
I20260926 02:59:41.842970 2875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42829 every 8 connection(s)
I20260926 02:59:41.844208 2809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:41.845359 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 2809
I20260926 02:59:41.845592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:41.846388 388 external_mini_cluster.cc:1428] Setting key 2a21542a788235b4e386ba9d10678342
I20260926 02:59:41.848778 2876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:59:41 dist-test-slave-fccw krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:41.861531 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0: Bootstrap starting.
I20260926 02:59:41.865582 2876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:41.866811 2876 log.cc:824] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:41.869155 2879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:41.848337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.97.62:42829)
0926 02:59:41.849132 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:41.849144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:41.849181 (+ 37us) server_negotiation.cc:407] Connection header received
0926 02:59:41.850830 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:41.850850 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:41.851215 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:41.851534 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:41.853310 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:41.854588 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:41.856324 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:41.856934 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:41.857127 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:41.861548 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:41.861559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:41.861577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:41.861625 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:41.865297 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:41.865963 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:41.865971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:41.865983 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:41.866177 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:41.866513 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:41.866518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:41.866521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:41.867130 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:41.867476 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:41.867816 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 02:59:41.868154 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":359,"threads_started":1}
I20260926 02:59:41.869743 2876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0: No bootstrap required, opened a new log
I20260926 02:59:41.877279 2876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260926 02:59:41.877625 2876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:41.877731 2876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569e0cb0ee9f4006a7c7a9ee41709de0, State: Initialized, Role: FOLLOWER
I20260926 02:59:41.878228 2876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260926 02:59:41.878451 2876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:41.878628 2876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:41.878829 2876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:41.880836 2876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260926 02:59:41.881309 2876 leader_election.cc:302] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 569e0cb0ee9f4006a7c7a9ee41709de0; no voters:
I20260926 02:59:41.881779 2876 leader_election.cc:288] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:41.881914 2881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:41.882936 2881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [term 1 LEADER]: Becoming Leader. State: Replica: 569e0cb0ee9f4006a7c7a9ee41709de0, State: Running, Role: LEADER
I20260926 02:59:41.883430 2881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } }
I20260926 02:59:41.883616 2876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:41.886145 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 569e0cb0ee9f4006a7c7a9ee41709de0. Latest consensus state: current_term: 1 leader_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } } }
I20260926 02:59:41.887018 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:41.886708 2882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "569e0cb0ee9f4006a7c7a9ee41709de0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42829 } } }
I20260926 02:59:41.887231 2882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:41.889117 2890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:41.893805 2890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:41.902150 2890 catalog_manager.cc:1347] Generated new cluster ID: 0a2e3020abf343dca33037ad656fbfd3
I20260926 02:59:41.902271 2890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:41.919438 2890 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:41.921355 2890 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:41.934527 2890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 569e0cb0ee9f4006a7c7a9ee41709de0: Generated new TSK 0
I20260926 02:59:41.935338 2890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:41.984349 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:42.304560 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:42.304826 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:42.304915 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:42.312345 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:42.312466 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:42.312650 2902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:42.322000 2902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:42.323199 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:42.324517 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:42.337307 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:42.337244 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:42.337275 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:42.337981 2902 server_base.cc:1034] running on GCE node
I20260926 02:59:42.338584 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:42.339418 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:42.340641 2902 hybrid_clock.cc:648] HybridClock initialized: now 1790391582340602 us; error 51 us; skew 500 ppm
I20260926 02:59:42.340932 2902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:42 dist-test-slave-fccw krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:42.343900 2902 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:42.345327 2902 webserver.cc:460] Webserver started at http://127.0.97.1:38675/ using document root <none> and password file <none>
I20260926 02:59:42.345985 2902 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:42.346123 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:42.346419 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:42.349233 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "433fd2ec0835477b89d4fde9f7246329"
format_stamp: "Formatted at 2026-09-26 02:59:42 on dist-test-slave-fccw"
server_key: "abead5b2b7d855d277f28c440aa720cd"
server_key_iv: "9ff8f264a9f85c04d82a6c4e53f3c85e"
server_key_version: "encryptionkey@0"
I20260926 02:59:42.349970 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "433fd2ec0835477b89d4fde9f7246329"
format_stamp: "Formatted at 2026-09-26 02:59:42 on dist-test-slave-fccw"
server_key: "abead5b2b7d855d277f28c440aa720cd"
server_key_iv: "9ff8f264a9f85c04d82a6c4e53f3c85e"
server_key_version: "encryptionkey@0"
I20260926 02:59:42.354784 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:59:42.357909 2917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:42.359169 2902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:59:42.359340 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "433fd2ec0835477b89d4fde9f7246329"
format_stamp: "Formatted at 2026-09-26 02:59:42 on dist-test-slave-fccw"
server_key: "abead5b2b7d855d277f28c440aa720cd"
server_key_iv: "9ff8f264a9f85c04d82a6c4e53f3c85e"
server_key_version: "encryptionkey@0"
I20260926 02:59:42.359563 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:42.378899 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:42.382624 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:42.382890 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:42.383826 2902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:42.385175 2902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:42.385274 2902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:42.385419 2902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:42.385493 2902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:42.417546 2902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45661
I20260926 02:59:42.417631 3029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45661 every 8 connection(s)
I20260926 02:59:42.418802 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:42.423096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 2902
I20260926 02:59:42.423316 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:42.424069 388 external_mini_cluster.cc:1428] Setting key 81c0ff989df27ff85dd8a66e208d0ae7
Sep 26 02:59:42 dist-test-slave-fccw krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:42.441221 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:42.421251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47339 (local address 127.0.97.62:42829)
0926 02:59:42.421679 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:42.421684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:42.422455 (+ 771us) server_negotiation.cc:407] Connection header received
0926 02:59:42.425048 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:42.425055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:42.425129 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:42.425217 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:42.427129 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.427835 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.430310 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.430701 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.430791 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:42.434351 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:42.434360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:42.434367 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:42.434410 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:42.438047 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:42.438715 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:42.438723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:42.438729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:42.438870 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:42.439307 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:42.439313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:42.439318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:42.439707 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:42.439872 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:42.440531 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 02:59:42.440779 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":250,"threads_started":1}
I20260926 02:59:42.441825 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:42.421643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42829 (local address 127.0.97.1:47339)
0926 02:59:42.422297 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:42.422338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:42.424712 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:42.425494 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:42.425505 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:42.426155 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:42.426942 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:42.426970 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.427996 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.428005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:42.430179 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:42.430195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.430918 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.430927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:42.431263 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:42.431984 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:42.432005 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:42.434138 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:42.438170 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:42.438176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:42.438189 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:42.438589 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:42.438983 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:42.438986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:42.438989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:42.439178 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:42.439831 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:42.439836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:42.440130 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 02:59:42.440371 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 02:59:42.440724 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":251,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 02:59:42.443405 3030 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42829
I20260926 02:59:42.443725 3030 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:42.444386 3030 heartbeater.cc:511] Master 127.0.97.62:42829 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 02:59:42.446333 2841 ts_manager.cc:194] Registered new tserver with Master: 433fd2ec0835477b89d4fde9f7246329 (127.0.97.1:45661)
I20260926 02:59:42.447880 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:42.466549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:42.769109 3036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:42.769372 3036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:42.769558 3036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:42.776463 3036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:42.776561 3036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:42.776718 3036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:42.786266 3036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:42.787606 3036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:42.788893 3036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:42.801157 3041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:42.801349 3044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:42.801414 3042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:42.802151 3036 server_base.cc:1034] running on GCE node
I20260926 02:59:42.802682 3036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:42.803558 3036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:42.804777 3036 hybrid_clock.cc:648] HybridClock initialized: now 1790391582804732 us; error 56 us; skew 500 ppm
I20260926 02:59:42.805120 3036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:42 dist-test-slave-fccw krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:42.816727 3036 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:42.818239 3036 webserver.cc:460] Webserver started at http://127.0.97.2:46251/ using document root <none> and password file <none>
I20260926 02:59:42.818878 3036 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:42.819013 3036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:42.819315 3036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:42.822184 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "f3c9a0bb0d6e4f0fb2138be5c6b55c9f"
format_stamp: "Formatted at 2026-09-26 02:59:42 on dist-test-slave-fccw"
server_key: "3a07a2b07a88c198de3cfe3866104b18"
server_key_iv: "ac2bc1d394b21608aa1bc47e5df25b2e"
server_key_version: "encryptionkey@0"
I20260926 02:59:42.822917 3036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f3c9a0bb0d6e4f0fb2138be5c6b55c9f"
format_stamp: "Formatted at 2026-09-26 02:59:42 on dist-test-slave-fccw"
server_key: "3a07a2b07a88c198de3cfe3866104b18"
server_key_iv: "ac2bc1d394b21608aa1bc47e5df25b2e"
server_key_version: "encryptionkey@0"
I20260926 02:59:42.827746 3036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 02:59:42.831765 3051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:42.832985 3036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 02:59:42.833226 3036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "f3c9a0bb0d6e4f0fb2138be5c6b55c9f"
format_stamp: "Formatted at 2026-09-26 02:59:42 on dist-test-slave-fccw"
server_key: "3a07a2b07a88c198de3cfe3866104b18"
server_key_iv: "ac2bc1d394b21608aa1bc47e5df25b2e"
server_key_version: "encryptionkey@0"
I20260926 02:59:42.833423 3036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:42.857448 3036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:42.861681 3036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:42.861949 3036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:42.862891 3036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:42.864283 3036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:42.864383 3036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:42.864521 3036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:42.864590 3036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:42.896117 3036 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37441
I20260926 02:59:42.896178 3163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37441 every 8 connection(s)
I20260926 02:59:42.897804 3036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:42.902465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3036
I20260926 02:59:42.902639 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:42.903085 388 external_mini_cluster.cc:1428] Setting key 102d889a50a2ebb2f416d4124c3a6132
Sep 26 02:59:42 dist-test-slave-fccw krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:42.918632 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:42.900789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47859 (local address 127.0.97.62:42829)
0926 02:59:42.900958 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:42.900962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:42.902092 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 02:59:42.904200 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:42.904204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:42.904277 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:42.904359 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:42.906046 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.907305 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.908927 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.909669 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.909802 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:42.913270 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:42.913275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:42.913278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:42.913306 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:42.915725 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:42.916433 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:42.916440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:42.916448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:42.916579 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:42.917025 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:42.917031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:42.917035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:42.917278 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:42.917425 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:42.918052 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 02:59:42.918282 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:59:42.919554 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:42.901276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42829 (local address 127.0.97.2:47859)
0926 02:59:42.901940 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:42.901978 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:42.903903 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:42.904620 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:42.904631 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:42.905331 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:42.905866 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:42.905883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.907429 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.907432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:42.908799 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:42.908811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:42.909850 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:42.909854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:42.910040 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:42.910690 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:42.910709 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:42.913078 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:42.915884 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:42.915891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:42.915924 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:42.916309 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:42.916684 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:42.916690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:42.916693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:42.916868 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:42.917409 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:42.917416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:42.917720 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 02:59:42.917962 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 02:59:42.918493 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 02:59:42.921047 3164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42829
I20260926 02:59:42.921388 3164 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:42.922096 3164 heartbeater.cc:511] Master 127.0.97.62:42829 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 02:59:42.923738 2841 ts_manager.cc:194] Registered new tserver with Master: f3c9a0bb0d6e4f0fb2138be5c6b55c9f (127.0.97.2:37441)
I20260926 02:59:42.925033 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:42.943934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:43.251170 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:43.251418 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:43.251579 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:43.258534 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:43.258749 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:43.258920 3169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:43.268121 3169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:43.269301 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:43.270498 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:43.283079 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:43.283183 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:43.283205 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:43.284029 3169 server_base.cc:1034] running on GCE node
I20260926 02:59:43.284590 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:43.285478 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:43.286700 3169 hybrid_clock.cc:648] HybridClock initialized: now 1790391583286656 us; error 56 us; skew 500 ppm
I20260926 02:59:43.286984 3169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:43 dist-test-slave-fccw krb5kdc[2796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:43.289999 3169 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:43.291167 3169 webserver.cc:460] Webserver started at http://127.0.97.3:40919/ using document root <none> and password file <none>
I20260926 02:59:43.291742 3169 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:43.291833 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:43.292061 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:43.294751 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "d7b6345e755b424cb4822b0e184f1bbc"
format_stamp: "Formatted at 2026-09-26 02:59:43 on dist-test-slave-fccw"
server_key: "f8b7013d393e8e0a4d6996e969611920"
server_key_iv: "c651ff6e28ae74f38d0897a148c18546"
server_key_version: "encryptionkey@0"
I20260926 02:59:43.295437 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d7b6345e755b424cb4822b0e184f1bbc"
format_stamp: "Formatted at 2026-09-26 02:59:43 on dist-test-slave-fccw"
server_key: "f8b7013d393e8e0a4d6996e969611920"
server_key_iv: "c651ff6e28ae74f38d0897a148c18546"
server_key_version: "encryptionkey@0"
I20260926 02:59:43.300030 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 02:59:43.303136 3184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:43.304322 3169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:43.304481 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "d7b6345e755b424cb4822b0e184f1bbc"
format_stamp: "Formatted at 2026-09-26 02:59:43 on dist-test-slave-fccw"
server_key: "f8b7013d393e8e0a4d6996e969611920"
server_key_iv: "c651ff6e28ae74f38d0897a148c18546"
server_key_version: "encryptionkey@0"
I20260926 02:59:43.304661 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:43.317309 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:43.320832 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:43.321121 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:43.321961 3169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:43.323148 3169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:43.323216 3169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:43.323377 3169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:43.323452 3169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:43.355203 3169 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39511
I20260926 02:59:43.355245 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39511 every 8 connection(s)
I20260926 02:59:43.356472 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:43.359486 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3169
I20260926 02:59:43.359691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:43.360252 388 external_mini_cluster.cc:1428] Setting key d29d2b171314a4206743bcc3434b330a
Sep 26 02:59:43 dist-test-slave-fccw krb5kdc[2796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:43.376477 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:43.358952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58633 (local address 127.0.97.62:42829)
0926 02:59:43.359122 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:43.359126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:43.360538 (+ 1412us) server_negotiation.cc:407] Connection header received
0926 02:59:43.362925 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:43.362929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:43.362990 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:43.363141 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:43.364952 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:43.365837 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:43.367276 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:43.367637 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:43.367725 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:43.370836 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:43.370840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:43.370844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:43.370872 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:43.373516 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:43.374285 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:43.374291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:43.374295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:43.374452 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:43.374850 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:43.374855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:43.374858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:43.375130 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:43.375280 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:43.375919 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 02:59:43.376137 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:59:43.377453 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:43.359556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42829 (local address 127.0.97.3:58633)
0926 02:59:43.360339 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:43.360397 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:43.362664 (+ 2267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:43.363557 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:43.363588 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:43.364248 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:43.364806 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:43.364822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:43.365969 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:43.365972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:43.367164 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:43.367175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:43.367805 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:43.367810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:43.367993 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:43.368610 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:43.368631 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:43.370671 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:43.373670 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:43.373676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:43.373688 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:43.374172 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:43.374550 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:43.374556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:43.374560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:43.374731 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:43.375256 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:43.375263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:43.375576 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 02:59:43.375820 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 02:59:43.376301 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":303,"threads_started":1}
I20260926 02:59:43.378710 3297 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42829
I20260926 02:59:43.378998 3297 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:43.379666 3297 heartbeater.cc:511] Master 127.0.97.62:42829 requested a full tablet report, sending...
I20260926 02:59:43.380964 2841 ts_manager.cc:194] Registered new tserver with Master: d7b6345e755b424cb4822b0e184f1bbc (127.0.97.3:39511)
I20260926 02:59:43.381765 2841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58633
I20260926 02:59:43.387535 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 02:59:43.416090 3032 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:59:43.413767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.0.97.62:42829)
0926 02:59:43.414032 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:43.414041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:43.414366 (+ 325us) server_negotiation.cc:407] Connection header received
0926 02:59:43.414520 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:43.414524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:43.414586 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:43.414672 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:43.415808 (+ 1136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53544: BlockingRecv error: recv got EOF from 127.0.0.1:53544 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 02:59:43.419100 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2902
W20260926 02:59:43.422245 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2902: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45661: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:59:43.422401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 2902
I20260926 02:59:43.434806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3036
W20260926 02:59:43.437285 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3036: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:37441: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:59:43.437446 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3036
I20260926 02:59:43.452504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3169
W20260926 02:59:43.455111 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3169: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39511: server requires authentication, but client does not have Kerberos credentials available
I20260926 02:59:43.455245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3169
I20260926 02:59:43.470026 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 2809
I20260926 02:59:43.471163 2871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:43.570633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 2809
2026-09-26T02:59:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3376 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:43 dist-test-slave-fccw krb5kdc[3314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:43 dist-test-slave-fccw krb5kdc[3314](info): set up 2 sockets
Sep 26 02:59:43 dist-test-slave-fccw krb5kdc[3314](info): commencing operation
krb5kdc: starting...
W20260926 02:59:44.653446 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:44 dist-test-slave-fccw krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:44.826520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33779
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:45.132079 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:45.132362 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:45.132565 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:45.139268 3327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:45.139395 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:45.139510 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:45.139602 3327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:45.139671 3327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:45.148748 3327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:45.149923 3327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:45.151173 3327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:45.162333 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:45.162364 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:45.162290 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:45.163268 3327 server_base.cc:1034] running on GCE node
I20260926 02:59:45.163960 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:45.165238 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:45.166469 3327 hybrid_clock.cc:648] HybridClock initialized: now 1790391585166435 us; error 45 us; skew 500 ppm
I20260926 02:59:45.166790 3327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:45 dist-test-slave-fccw krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:45.170006 3327 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:45.171433 3327 webserver.cc:460] Webserver started at http://127.0.97.62:43831/ using document root <none> and password file <none>
I20260926 02:59:45.172088 3327 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:45.172223 3327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:45.172506 3327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:45.175333 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "cdf363147fa84fd4911a4f0422370963"
format_stamp: "Formatted at 2026-09-26 02:59:45 on dist-test-slave-fccw"
server_key: "13413ea3844ca51b3acd61a2bf8d7e05"
server_key_iv: "8b6c19a13a283b1d0b4701210adfc057"
server_key_version: "encryptionkey@0"
I20260926 02:59:45.176095 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "cdf363147fa84fd4911a4f0422370963"
format_stamp: "Formatted at 2026-09-26 02:59:45 on dist-test-slave-fccw"
server_key: "13413ea3844ca51b3acd61a2bf8d7e05"
server_key_iv: "8b6c19a13a283b1d0b4701210adfc057"
server_key_version: "encryptionkey@0"
I20260926 02:59:45.180859 3327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 02:59:45.184000 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:45.185227 3327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:45.185398 3327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "cdf363147fa84fd4911a4f0422370963"
format_stamp: "Formatted at 2026-09-26 02:59:45 on dist-test-slave-fccw"
server_key: "13413ea3844ca51b3acd61a2bf8d7e05"
server_key_iv: "8b6c19a13a283b1d0b4701210adfc057"
server_key_version: "encryptionkey@0"
I20260926 02:59:45.185614 3327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:45.197721 3327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:45.201999 3327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:45.202276 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:45.219497 3327 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33779
I20260926 02:59:45.219532 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33779 every 8 connection(s)
I20260926 02:59:45.220824 3327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:45.221565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3327
I20260926 02:59:45.221763 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:45.222463 388 external_mini_cluster.cc:1428] Setting key 396b1489ae668f3110e74b8895a7542f
I20260926 02:59:45.225126 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 02:59:45 dist-test-slave-fccw krb5kdc[3314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:45.235411 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963: Bootstrap starting.
I20260926 02:59:45.238897 3394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:45.240090 3394 log.cc:824] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:45.243145 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963: No bootstrap required, opened a new log
I20260926 02:59:45.243587 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:45.224272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.0.97.62:33779)
0926 02:59:45.224989 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:45.225046 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:45.225094 (+ 48us) server_negotiation.cc:407] Connection header received
0926 02:59:45.226629 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:45.226649 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:45.226995 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:45.227399 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:45.228848 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:45.230248 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:45.231468 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:45.232120 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:45.232332 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:45.236516 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:45.236523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:45.236536 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:45.236571 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:45.239489 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:45.240277 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:45.240286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:45.240298 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:45.240451 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:45.240896 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:45.240902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:45.240905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:45.241553 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:45.241757 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:45.242256 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 02:59:45.242585 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":330,"threads_started":1}
I20260926 02:59:45.247406 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:59:45.253789 3394 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf363147fa84fd4911a4f0422370963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260926 02:59:45.254209 3394 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:45.254325 3394 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf363147fa84fd4911a4f0422370963, State: Initialized, Role: FOLLOWER
I20260926 02:59:45.254951 3394 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf363147fa84fd4911a4f0422370963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260926 02:59:45.255228 3394 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:45.255358 3394 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:45.255533 3394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:45.257710 3394 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf363147fa84fd4911a4f0422370963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260926 02:59:45.258340 3394 leader_election.cc:302] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdf363147fa84fd4911a4f0422370963; no voters:
I20260926 02:59:45.259066 3394 leader_election.cc:288] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:45.259233 3400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:45.261669 3394 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:45.262996 3400 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [term 1 LEADER]: Becoming Leader. State: Replica: cdf363147fa84fd4911a4f0422370963, State: Running, Role: LEADER
I20260926 02:59:45.263763 3400 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf363147fa84fd4911a4f0422370963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } }
I20260926 02:59:45.266963 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdf363147fa84fd4911a4f0422370963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf363147fa84fd4911a4f0422370963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } } }
I20260926 02:59:45.267326 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:45.267059 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdf363147fa84fd4911a4f0422370963. Latest consensus state: current_term: 1 leader_uuid: "cdf363147fa84fd4911a4f0422370963" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf363147fa84fd4911a4f0422370963" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33779 } } }
I20260926 02:59:45.267570 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:45.268882 3411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:45.277891 3411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:45.285115 3418 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 02:59:45.288168 3411 catalog_manager.cc:1347] Generated new cluster ID: 18b4acb3c38b44168b4b255f147a4fa4
I20260926 02:59:45.288338 3411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:45.301415 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:45.288182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.0.97.62:33779)
0926 02:59:45.288359 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:45.288365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:45.293621 (+ 5256us) server_negotiation.cc:407] Connection header received
0926 02:59:45.293794 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:45.293800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:45.293874 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:45.293971 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:45.295158 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:45.296209 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:45.298007 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:45.298601 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:45.298704 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:45.299856 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:45.299864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:45.299869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:45.300614 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:45.300662 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:45.300685 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:45.300774 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:45.300944 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 02:59:45.301044 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:59:45.307672 3411 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:45.309348 3411 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:45.316725 3411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdf363147fa84fd4911a4f0422370963: Generated new TSK 0
I20260926 02:59:45.317687 3411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:59:45.326309 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3327
I20260926 02:59:45.327373 3389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:45.413369 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3327
2026-09-26T02:59:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1836 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:45 dist-test-slave-fccw krb5kdc[3428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:45 dist-test-slave-fccw krb5kdc[3428](info): set up 2 sockets
Sep 26 02:59:45 dist-test-slave-fccw krb5kdc[3428](info): commencing operation
krb5kdc: starting...
W20260926 02:59:46.457394 388 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:46 dist-test-slave-fccw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:46.644743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46315
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:46.951308 3441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:46.951560 3441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:46.951638 3441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:46.958375 3441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:46.958463 3441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:46.958508 3441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:46.958593 3441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:46.958657 3441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:46.968060 3441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:46.969343 3441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:46.970573 3441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:46.981976 3449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:46.982046 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:46.982415 3447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:46.982782 3441 server_base.cc:1034] running on GCE node
I20260926 02:59:46.983480 3441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:46.984737 3441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:46.985967 3441 hybrid_clock.cc:648] HybridClock initialized: now 1790391586985940 us; error 43 us; skew 500 ppm
I20260926 02:59:46.986315 3441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:46 dist-test-slave-fccw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:46.989174 3441 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:46.990543 3441 webserver.cc:460] Webserver started at http://127.0.97.62:41803/ using document root <none> and password file <none>
I20260926 02:59:46.991199 3441 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:46.991351 3441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:46.991655 3441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:46.994549 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "e0d4b759705f416491766cb2fe0cf25e"
format_stamp: "Formatted at 2026-09-26 02:59:46 on dist-test-slave-fccw"
server_key: "cecd065b692089f1f24e87051b14ed94"
server_key_iv: "e48af8b98c1bd0a2c883465653247083"
server_key_version: "encryptionkey@0"
I20260926 02:59:46.995261 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "e0d4b759705f416491766cb2fe0cf25e"
format_stamp: "Formatted at 2026-09-26 02:59:46 on dist-test-slave-fccw"
server_key: "cecd065b692089f1f24e87051b14ed94"
server_key_iv: "e48af8b98c1bd0a2c883465653247083"
server_key_version: "encryptionkey@0"
I20260926 02:59:47.000221 3441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:59:47.003337 3456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:47.004510 3441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:47.004670 3441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "e0d4b759705f416491766cb2fe0cf25e"
format_stamp: "Formatted at 2026-09-26 02:59:46 on dist-test-slave-fccw"
server_key: "cecd065b692089f1f24e87051b14ed94"
server_key_iv: "e48af8b98c1bd0a2c883465653247083"
server_key_version: "encryptionkey@0"
I20260926 02:59:47.004845 3441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:47.017498 3441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:47.021076 3441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:47.021330 3441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:47.038318 3441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46315
I20260926 02:59:47.038339 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46315 every 8 connection(s)
I20260926 02:59:47.039626 3441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:47.040606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3441
I20260926 02:59:47.040818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:47.041631 388 external_mini_cluster.cc:1428] Setting key e4e72c71430aa3dbd864ad2f313ec7be
I20260926 02:59:47.044119 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:47.054302 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e: Bootstrap starting.
Sep 26 02:59:47 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:47.057829 3508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:47.058856 3508 log.cc:824] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:47.062007 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e: No bootstrap required, opened a new log
I20260926 02:59:47.062742 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:47.043592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.97.62:46315)
0926 02:59:47.044344 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:47.044355 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:47.044398 (+ 43us) server_negotiation.cc:407] Connection header received
0926 02:59:47.046011 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:47.046031 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:47.046386 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:47.046773 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:47.048182 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:47.049518 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:47.050716 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:47.051357 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:47.051563 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:47.055758 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:47.055765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:47.055781 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:47.055812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:47.058755 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:47.059594 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:47.059603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:47.059610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:47.059809 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:47.060131 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:47.060138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:47.060144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:47.060649 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:47.060855 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:47.061310 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 02:59:47.061643 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260926 02:59:47.069906 3508 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d4b759705f416491766cb2fe0cf25e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46315 } }
I20260926 02:59:47.070250 3508 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:47.070398 3508 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d4b759705f416491766cb2fe0cf25e, State: Initialized, Role: FOLLOWER
I20260926 02:59:47.070955 3508 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d4b759705f416491766cb2fe0cf25e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46315 } }
I20260926 02:59:47.071174 3508 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:47.071308 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:47.071491 3508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:47.073576 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d4b759705f416491766cb2fe0cf25e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46315 } }
I20260926 02:59:47.074069 3508 leader_election.cc:302] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d4b759705f416491766cb2fe0cf25e; no voters:
I20260926 02:59:47.074677 3508 leader_election.cc:288] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:47.074795 3513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:47.075482 3513 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [term 1 LEADER]: Becoming Leader. State: Replica: e0d4b759705f416491766cb2fe0cf25e, State: Running, Role: LEADER
I20260926 02:59:47.076066 3513 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d4b759705f416491766cb2fe0cf25e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46315 } }
I20260926 02:59:47.076617 3508 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:47.078994 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d4b759705f416491766cb2fe0cf25e. Latest consensus state: current_term: 1 leader_uuid: "e0d4b759705f416491766cb2fe0cf25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d4b759705f416491766cb2fe0cf25e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46315 } } }
I20260926 02:59:47.079291 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:47.079763 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d4b759705f416491766cb2fe0cf25e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d4b759705f416491766cb2fe0cf25e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46315 } } }
I20260926 02:59:47.080067 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:47.081688 3520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:47.086476 3520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:47.099650 3520 catalog_manager.cc:1347] Generated new cluster ID: 21a7b60cc0e74be7bdbe4b918ef31bbc
I20260926 02:59:47.099767 3520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:47.113970 3520 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:47.115631 3520 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:47.126818 3520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d4b759705f416491766cb2fe0cf25e: Generated new TSK 0
I20260926 02:59:47.127769 3520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:47.171612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46315
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:47.482468 3534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:47.482720 3534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:47.482816 3534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:47.489734 3534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:47.489828 3534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:47.489931 3534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:47.499089 3534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:47.500259 3534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:47.501500 3534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:47.513871 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:47.513916 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:47.513795 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:47.514822 3534 server_base.cc:1034] running on GCE node
I20260926 02:59:47.515413 3534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:47.516336 3534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:47.517558 3534 hybrid_clock.cc:648] HybridClock initialized: now 1790391587517472 us; error 71 us; skew 500 ppm
I20260926 02:59:47.517874 3534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:47 dist-test-slave-fccw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:47.520812 3534 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:47.522256 3534 webserver.cc:460] Webserver started at http://127.0.97.1:42757/ using document root <none> and password file <none>
I20260926 02:59:47.522929 3534 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:47.523068 3534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:47.523361 3534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:47.526216 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "622f9900310e422cbc2f6a3086a15968"
format_stamp: "Formatted at 2026-09-26 02:59:47 on dist-test-slave-fccw"
server_key: "8c785948eb83eb7a711a3b0677a126aa"
server_key_iv: "f087d96eef858448abccdd1e14e1f361"
server_key_version: "encryptionkey@0"
I20260926 02:59:47.527045 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "622f9900310e422cbc2f6a3086a15968"
format_stamp: "Formatted at 2026-09-26 02:59:47 on dist-test-slave-fccw"
server_key: "8c785948eb83eb7a711a3b0677a126aa"
server_key_iv: "f087d96eef858448abccdd1e14e1f361"
server_key_version: "encryptionkey@0"
I20260926 02:59:47.531843 3534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 02:59:47.535060 3549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:47.536267 3534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 02:59:47.536435 3534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "622f9900310e422cbc2f6a3086a15968"
format_stamp: "Formatted at 2026-09-26 02:59:47 on dist-test-slave-fccw"
server_key: "8c785948eb83eb7a711a3b0677a126aa"
server_key_iv: "f087d96eef858448abccdd1e14e1f361"
server_key_version: "encryptionkey@0"
I20260926 02:59:47.536656 3534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:47.562819 3534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:47.566906 3534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:47.567204 3534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:47.568099 3534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:47.569376 3534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:47.569469 3534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:47.569550 3534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:47.569591 3534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:47.601519 3534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37767
I20260926 02:59:47.601560 3661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37767 every 8 connection(s)
I20260926 02:59:47.602811 3534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:47.607993 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3534
I20260926 02:59:47.608184 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:47.608819 388 external_mini_cluster.cc:1428] Setting key a6527362c1a9c1505b30112c5d8b0c80
Sep 26 02:59:47 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:47.623210 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:47.605391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56321 (local address 127.0.97.62:46315)
0926 02:59:47.605789 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:47.605794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:47.606797 (+ 1003us) server_negotiation.cc:407] Connection header received
0926 02:59:47.608887 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:47.608894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:47.608981 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:47.609118 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:47.610793 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:47.611904 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:47.613465 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:47.614184 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:47.614327 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:47.617936 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:47.617940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:47.617944 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:47.617972 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:47.620415 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:47.621107 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:47.621114 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:47.621118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:47.621235 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:47.621653 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:47.621658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:47.621661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:47.621923 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:47.622085 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:47.622672 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 02:59:47.622867 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":212,"threads_started":1}
I20260926 02:59:47.624167 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:47.605983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46315 (local address 127.0.97.1:56321)
0926 02:59:47.606645 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:47.606687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:47.608569 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:47.609368 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:47.609384 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:47.609990 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:47.610639 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:47.610658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:47.612038 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:47.612041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:47.613334 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:47.613348 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:47.614399 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:47.614404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:47.614591 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:47.615247 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:47.615266 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:47.617714 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:47.620549 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:47.620557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:47.620572 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:47.620966 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:47.621340 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:47.621346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:47.621351 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:47.621527 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:47.622053 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:47.622062 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:47.622354 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 02:59:47.622589 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 02:59:47.623075 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":265,"threads_started":1}
I20260926 02:59:47.625664 3662 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46315
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 02:59:47.626008 3662 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:47.626693 3662 heartbeater.cc:511] Master 127.0.97.62:46315 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 02:59:47.628952 3473 ts_manager.cc:194] Registered new tserver with Master: 622f9900310e422cbc2f6a3086a15968 (127.0.97.1:37767)
I20260926 02:59:47.630450 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:47.650159 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46315
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:47.957716 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:47.957980 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:47.958068 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:47.964986 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:47.965092 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:47.965191 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:47.974264 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:47.975426 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:47.976755 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:47.989542 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:47.989516 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:47.989504 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:47.990059 3668 server_base.cc:1034] running on GCE node
I20260926 02:59:47.990607 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:47.991369 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:47.992583 3668 hybrid_clock.cc:648] HybridClock initialized: now 1790391587992535 us; error 58 us; skew 500 ppm
I20260926 02:59:47.992877 3668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:47 dist-test-slave-fccw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:48.000850 3668 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:48.002311 3668 webserver.cc:460] Webserver started at http://127.0.97.2:37287/ using document root <none> and password file <none>
I20260926 02:59:48.002964 3668 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:48.003099 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:48.003396 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:48.006294 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "52212558d04943f4b88895bfc6b45cc8"
format_stamp: "Formatted at 2026-09-26 02:59:47 on dist-test-slave-fccw"
server_key: "14594470b36d9ffee94451c74cd3e9cd"
server_key_iv: "1fa18c9ec70aa71ef4c7d2b707a9112d"
server_key_version: "encryptionkey@0"
I20260926 02:59:48.007081 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "52212558d04943f4b88895bfc6b45cc8"
format_stamp: "Formatted at 2026-09-26 02:59:47 on dist-test-slave-fccw"
server_key: "14594470b36d9ffee94451c74cd3e9cd"
server_key_iv: "1fa18c9ec70aa71ef4c7d2b707a9112d"
server_key_version: "encryptionkey@0"
I20260926 02:59:48.012086 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 02:59:48.015254 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:48.016463 3668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:48.016631 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "52212558d04943f4b88895bfc6b45cc8"
format_stamp: "Formatted at 2026-09-26 02:59:47 on dist-test-slave-fccw"
server_key: "14594470b36d9ffee94451c74cd3e9cd"
server_key_iv: "1fa18c9ec70aa71ef4c7d2b707a9112d"
server_key_version: "encryptionkey@0"
I20260926 02:59:48.016806 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:48.040102 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:48.044088 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:48.044373 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:48.045444 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:48.046804 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:48.046914 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:48.047039 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:48.047113 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:48.079633 3668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43305
I20260926 02:59:48.079667 3795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43305 every 8 connection(s)
I20260926 02:59:48.080816 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:48.085875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3668
I20260926 02:59:48.086037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:48.086611 388 external_mini_cluster.cc:1428] Setting key 3e736e5a9947b5d4c36e7bed66f9c3e7
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:48.102988 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.083387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46017 (local address 127.0.97.62:46315)
0926 02:59:48.083552 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:48.083556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:48.084631 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 02:59:48.087058 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:48.087062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:48.087125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:48.087252 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:48.088868 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.089625 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.092067 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.092605 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.092694 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.096188 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:48.096197 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:48.096205 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:48.096246 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:48.099762 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.100440 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.100448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.100454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.100617 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.101018 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.101025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.101030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.101386 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:48.101579 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:48.102270 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 02:59:48.102534 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:59:48.103580 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.083842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46315 (local address 127.0.97.2:46017)
0926 02:59:48.084485 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:48.084520 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:48.086756 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:48.087487 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:48.087497 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:48.088023 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:48.088714 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:48.088737 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.089773 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.089781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:48.091931 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:48.091947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.092823 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.092839 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:48.093168 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.093870 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:48.093905 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:48.095993 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:48.099889 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:48.099895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:48.099907 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:48.100317 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:48.100724 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:48.100727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:48.100729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:48.100892 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:48.101571 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:48.101580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:48.101919 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 02:59:48.102138 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 02:59:48.102491 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":251,"threads_started":1}
I20260926 02:59:48.105054 3796 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46315
I20260926 02:59:48.105348 3796 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 02:59:48.106069 3796 heartbeater.cc:511] Master 127.0.97.62:46315 requested a full tablet report, sending...
I20260926 02:59:48.107571 3473 ts_manager.cc:194] Registered new tserver with Master: 52212558d04943f4b88895bfc6b45cc8 (127.0.97.2:43305)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 02:59:48.108611 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:48.131426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46315
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:48.441804 3801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:48.442077 3801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:48.442250 3801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:48.449210 3801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:48.449327 3801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:48.449537 3801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:48.458807 3801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:48.460043 3801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:48.461334 3801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:48.474073 3807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:48.474187 3809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:48.474044 3806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:48.475090 3801 server_base.cc:1034] running on GCE node
I20260926 02:59:48.475660 3801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:48.476583 3801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:48.477798 3801 hybrid_clock.cc:648] HybridClock initialized: now 1790391588477738 us; error 62 us; skew 500 ppm
I20260926 02:59:48.478123 3801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:48.481137 3801 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:48.482555 3801 webserver.cc:460] Webserver started at http://127.0.97.3:35203/ using document root <none> and password file <none>
I20260926 02:59:48.483290 3801 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:48.483429 3801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:48.483709 3801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:48.486536 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "704fdbc8fbe04d14b3bc59b812c5db6c"
format_stamp: "Formatted at 2026-09-26 02:59:48 on dist-test-slave-fccw"
server_key: "2e47094abed961e99269f806a542d351"
server_key_iv: "22c78d9f9b1f6c7013a00d12675d13bd"
server_key_version: "encryptionkey@0"
I20260926 02:59:48.487341 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "704fdbc8fbe04d14b3bc59b812c5db6c"
format_stamp: "Formatted at 2026-09-26 02:59:48 on dist-test-slave-fccw"
server_key: "2e47094abed961e99269f806a542d351"
server_key_iv: "22c78d9f9b1f6c7013a00d12675d13bd"
server_key_version: "encryptionkey@0"
I20260926 02:59:48.492089 3801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 02:59:48.495251 3816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:48.496456 3801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 02:59:48.496624 3801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "704fdbc8fbe04d14b3bc59b812c5db6c"
format_stamp: "Formatted at 2026-09-26 02:59:48 on dist-test-slave-fccw"
server_key: "2e47094abed961e99269f806a542d351"
server_key_iv: "22c78d9f9b1f6c7013a00d12675d13bd"
server_key_version: "encryptionkey@0"
I20260926 02:59:48.496804 3801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:48.526414 3801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:48.530866 3801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:48.531194 3801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:48.532177 3801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:48.533532 3801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:48.533630 3801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:48.533779 3801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:48.533859 3801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:48.565608 3801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42855
I20260926 02:59:48.565652 3928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42855 every 8 connection(s)
I20260926 02:59:48.566962 3801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:48.568634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3801
I20260926 02:59:48.568809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:48.569355 388 external_mini_cluster.cc:1428] Setting key 046d236094f34bc3b843d22c8f68f97b
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:48.586588 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.569551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60845 (local address 127.0.97.62:46315)
0926 02:59:48.569773 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:48.569779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:48.571037 (+ 1258us) server_negotiation.cc:407] Connection header received
0926 02:59:48.572962 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:48.572967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:48.573067 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:48.573177 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:48.574562 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.575295 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.576771 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.577332 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.577457 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.580896 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:48.580901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:48.580904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:48.580934 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:48.584058 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.584762 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.584766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.584768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.584874 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.585288 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.585291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.585293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.585535 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:48.585628 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:48.586212 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 02:59:48.586355 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 02:59:48.587481 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.570145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46315 (local address 127.0.97.3:60845)
0926 02:59:48.570892 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:48.570933 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:48.572717 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:48.573411 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:48.573420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:48.573877 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:48.574405 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:48.574421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.575445 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.575451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:48.576633 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:48.576644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.577548 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.577555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:48.577783 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.578460 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:48.578480 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:48.580713 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:48.584223 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:48.584232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:48.584247 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:48.584665 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:48.584990 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:48.585031 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:48.585035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:48.585203 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:48.585635 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:48.585640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:48.585912 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 02:59:48.586132 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 02:59:48.586487 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":302,"threads_started":1}
I20260926 02:59:48.588686 3929 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46315
I20260926 02:59:48.588973 3929 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:48.589695 3929 heartbeater.cc:511] Master 127.0.97.62:46315 requested a full tablet report, sending...
I20260926 02:59:48.590945 3473 ts_manager.cc:194] Registered new tserver with Master: 704fdbc8fbe04d14b3bc59b812c5db6c (127.0.97.3:42855)
I20260926 02:59:48.591683 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60845
I20260926 02:59:48.596171 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391588, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260926 02:59:48.633319 3662 heartbeater.cc:503] Master 127.0.97.62:46315 was elected leader, sending a full tablet report...
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391588, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:48.640750 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.627816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.0.97.62:46315)
0926 02:59:48.627988 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:48.627992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:48.628418 (+ 426us) server_negotiation.cc:407] Connection header received
0926 02:59:48.628554 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:48.628558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:48.628627 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:48.628708 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:48.629965 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.630625 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.631742 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.632106 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.632189 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.636101 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:48.636105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:48.636109 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:48.636136 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:48.638430 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.639014 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.639018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.639020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.639131 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.639611 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.639614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.639616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.639878 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:48.640059 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:48.640331 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 02:59:48.640538 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 02:59:48.641227 3473 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39438
I20260926 02:59:48.642829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3534
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391588, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:48.656915 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.643805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.0.97.1:37767)
0926 02:59:48.644364 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:48.644370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:48.644392 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:59:48.644438 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:48.644444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:48.644652 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:48.644806 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:48.645845 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.646819 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.647962 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.648493 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.648624 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.652243 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:48.652253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:48.652266 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:48.652297 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:48.654830 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.655416 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.655422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.655424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.655535 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.655811 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.655814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.655815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.656094 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:48.656286 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:48.656488 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 02:59:48.656659 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":285,"threads_started":1}
W20260926 02:59:48.658107 3657 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42830
W20260926 02:59:48.659157 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:48.659363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3534
I20260926 02:59:48.674746 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3668
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391588, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 02:59:48.689174 3941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.675484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.0.97.2:43305)
0926 02:59:48.675963 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:48.675969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:48.675993 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:59:48.676046 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:48.676052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:48.676288 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:48.676432 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:48.677726 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.678680 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.679801 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.680324 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.680415 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.684321 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:48.684332 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:48.684344 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:48.684379 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:48.686917 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.687458 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.687464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.687467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.687580 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.687885 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.687888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.687889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.688203 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:48.688409 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:48.688666 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 02:59:48.688849 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":254,"threads_started":1}
W20260926 02:59:48.690382 3791 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60464
W20260926 02:59:48.691357 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:48.691498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3668
I20260926 02:59:48.706374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3801
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391588, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 02:59:48.721596 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:48.707183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.0.97.3:42855)
0926 02:59:48.707756 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:48.707762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:48.707788 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:59:48.707839 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:48.707845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:48.708038 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:48.708202 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:48.709635 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.710620 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.711801 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:48.712334 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:48.712480 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:48.716513 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:48.716522 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:48.716534 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:48.716562 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:48.719001 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.719612 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.719619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.719621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.719723 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:48.720062 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:48.720067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:48.720070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:48.720413 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:48.720707 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:48.720981 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 02:59:48.721255 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":325,"threads_started":1}
W20260926 02:59:48.722709 3924 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51102
W20260926 02:59:48.723762 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 02:59:48.723932 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3801
I20260926 02:59:48.738498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3441
I20260926 02:59:48.739478 3503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:48.842623 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3441
2026-09-26T02:59:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3434 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3946](info): set up 2 sockets
Sep 26 02:59:48 dist-test-slave-fccw krb5kdc[3946](info): commencing operation
krb5kdc: starting...
W20260926 02:59:49.917416 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:50 dist-test-slave-fccw krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:50.096276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38553
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:50.399125 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:50.399389 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:50.399468 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:50.406376 3959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:50.406468 3959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:50.406512 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:50.406591 3959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:50.406656 3959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:50.415822 3959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:50.417119 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:50.418347 3959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:50.429605 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:50.429682 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:50.429656 3965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:50.430392 3959 server_base.cc:1034] running on GCE node
I20260926 02:59:50.431128 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:50.432412 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:50.433630 3959 hybrid_clock.cc:648] HybridClock initialized: now 1790391590433597 us; error 45 us; skew 500 ppm
I20260926 02:59:50.433995 3959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:50 dist-test-slave-fccw krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:50.436750 3959 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:50.438021 3959 webserver.cc:460] Webserver started at http://127.0.97.62:40805/ using document root <none> and password file <none>
I20260926 02:59:50.438578 3959 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:50.438668 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:50.438935 3959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:50.441815 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "e9d8622071484a92a2f9346c74fab44d"
format_stamp: "Formatted at 2026-09-26 02:59:50 on dist-test-slave-fccw"
server_key: "ae4c80022df35db4249a083124af1cc2"
server_key_iv: "a076e684d413644c89dd0e085331c6da"
server_key_version: "encryptionkey@0"
I20260926 02:59:50.442487 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "e9d8622071484a92a2f9346c74fab44d"
format_stamp: "Formatted at 2026-09-26 02:59:50 on dist-test-slave-fccw"
server_key: "ae4c80022df35db4249a083124af1cc2"
server_key_iv: "a076e684d413644c89dd0e085331c6da"
server_key_version: "encryptionkey@0"
I20260926 02:59:50.447144 3959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:59:50.450199 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:50.451421 3959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:50.451588 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "e9d8622071484a92a2f9346c74fab44d"
format_stamp: "Formatted at 2026-09-26 02:59:50 on dist-test-slave-fccw"
server_key: "ae4c80022df35db4249a083124af1cc2"
server_key_iv: "a076e684d413644c89dd0e085331c6da"
server_key_version: "encryptionkey@0"
I20260926 02:59:50.451807 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:50.480453 3959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:50.484398 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:50.484670 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:50.502542 3959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38553
I20260926 02:59:50.502547 4025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38553 every 8 connection(s)
I20260926 02:59:50.503896 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:50.507411 4026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:50.512899 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 3959
I20260926 02:59:50.513168 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:50.513780 388 external_mini_cluster.cc:1428] Setting key 8466aa2807d9779e0eb0221b0e8536e8
I20260926 02:59:50.517582 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d: Bootstrap starting.
I20260926 02:59:50.521288 4026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:50.522403 4026 log.cc:824] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:50.525308 4026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d: No bootstrap required, opened a new log
Sep 26 02:59:50 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:50.533157 4026 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8622071484a92a2f9346c74fab44d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38553 } }
I20260926 02:59:50.533519 4026 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:50.533636 4026 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d8622071484a92a2f9346c74fab44d, State: Initialized, Role: FOLLOWER
I20260926 02:59:50.534310 4026 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8622071484a92a2f9346c74fab44d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38553 } }
I20260926 02:59:50.534606 4026 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:50.534749 4026 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:50.534979 4026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:50.535988 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:50.515651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.0.97.62:38553)
0926 02:59:50.516400 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:50.516411 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:50.516451 (+ 40us) server_negotiation.cc:407] Connection header received
0926 02:59:50.518155 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:50.518176 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:50.518568 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:50.518873 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:50.520680 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:50.522065 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:50.523614 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:50.524233 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:50.524355 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:50.528734 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:50.528741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:50.528757 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:50.528793 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:50.531878 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:50.532552 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:50.532560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:50.532574 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:50.532726 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:50.533122 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:50.533128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:50.533133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:50.533710 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:50.533989 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:50.534483 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 02:59:50.534961 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":361,"threads_started":1}
I20260926 02:59:50.537384 4026 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8622071484a92a2f9346c74fab44d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38553 } }
I20260926 02:59:50.537881 4026 leader_election.cc:302] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d8622071484a92a2f9346c74fab44d; no voters:
I20260926 02:59:50.538515 4026 leader_election.cc:288] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:50.538650 4031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:50.539377 4031 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [term 1 LEADER]: Becoming Leader. State: Replica: e9d8622071484a92a2f9346c74fab44d, State: Running, Role: LEADER
I20260926 02:59:50.539876 4031 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8622071484a92a2f9346c74fab44d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38553 } }
I20260926 02:59:50.540553 4026 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:50.542567 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d8622071484a92a2f9346c74fab44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8622071484a92a2f9346c74fab44d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38553 } } }
I20260926 02:59:50.542577 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d8622071484a92a2f9346c74fab44d. Latest consensus state: current_term: 1 leader_uuid: "e9d8622071484a92a2f9346c74fab44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d8622071484a92a2f9346c74fab44d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38553 } } }
I20260926 02:59:50.542999 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:50.542999 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:50.544628 4036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:50.553774 4036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:50.561381 4036 catalog_manager.cc:1347] Generated new cluster ID: c1ffe96058f3429a92bc9ec02a098d5a
I20260926 02:59:50.561486 4036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:50.572113 4036 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:50.573621 4036 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:50.583173 4036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d8622071484a92a2f9346c74fab44d: Generated new TSK 0
I20260926 02:59:50.584005 4036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:50.632301 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38553
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:50.941176 4052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:50.941455 4052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:50.941548 4052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:50.948472 4052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:50.948575 4052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:50.948676 4052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:50.958527 4052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:50.959728 4052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:50.960970 4052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:50.973594 4057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:50.973668 4060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:50.973609 4058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:50.975075 4052 server_base.cc:1034] running on GCE node
I20260926 02:59:50.975608 4052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:50.976455 4052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:50.977650 4052 hybrid_clock.cc:648] HybridClock initialized: now 1790391590977588 us; error 51 us; skew 500 ppm
I20260926 02:59:50.977937 4052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:50 dist-test-slave-fccw krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:50.980844 4052 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:50.982230 4052 webserver.cc:460] Webserver started at http://127.0.97.1:36259/ using document root <none> and password file <none>
I20260926 02:59:50.982936 4052 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:50.983031 4052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:50.983266 4052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:50.986024 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "d93e557dc49a4fa89ff20ba23b9ddfb1"
format_stamp: "Formatted at 2026-09-26 02:59:50 on dist-test-slave-fccw"
server_key: "e4932dd5a8aa3abb6f2bf7e472328469"
server_key_iv: "6fe67ddefca8f26200eb06ff0d7dae33"
server_key_version: "encryptionkey@0"
I20260926 02:59:50.986742 4052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d93e557dc49a4fa89ff20ba23b9ddfb1"
format_stamp: "Formatted at 2026-09-26 02:59:50 on dist-test-slave-fccw"
server_key: "e4932dd5a8aa3abb6f2bf7e472328469"
server_key_iv: "6fe67ddefca8f26200eb06ff0d7dae33"
server_key_version: "encryptionkey@0"
I20260926 02:59:50.991583 4052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 02:59:50.994642 4067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:50.995842 4052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 02:59:50.995965 4052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "d93e557dc49a4fa89ff20ba23b9ddfb1"
format_stamp: "Formatted at 2026-09-26 02:59:50 on dist-test-slave-fccw"
server_key: "e4932dd5a8aa3abb6f2bf7e472328469"
server_key_iv: "6fe67ddefca8f26200eb06ff0d7dae33"
server_key_version: "encryptionkey@0"
I20260926 02:59:50.996106 4052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:51.007001 4052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:51.010914 4052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:51.011142 4052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:51.011929 4052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:51.013106 4052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:51.013173 4052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.013250 4052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:51.013298 4052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.045154 4052 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39847
I20260926 02:59:51.045212 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39847 every 8 connection(s)
I20260926 02:59:51.046397 4052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:51.048516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 4052
I20260926 02:59:51.048715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:51.049530 388 external_mini_cluster.cc:1428] Setting key ceb907ff828010914501ddce5818ae43
Sep 26 02:59:51 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:51.069380 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:51.048909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49419 (local address 127.0.97.62:38553)
0926 02:59:51.049398 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:51.049404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:51.053095 (+ 3691us) server_negotiation.cc:407] Connection header received
0926 02:59:51.053167 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:51.053171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:51.053228 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:51.053317 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:51.055382 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.056187 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.058206 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.058595 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.058728 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:51.062693 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:51.062703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:51.062711 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:51.062754 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:51.066321 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:51.066975 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:51.066983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:51.066989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:51.067135 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:51.067533 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:51.067540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:51.067546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:51.067934 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:51.068085 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:51.068682 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 02:59:51.068916 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260926 02:59:51.070024 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:51.049555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38553 (local address 127.0.97.1:49419)
0926 02:59:51.050447 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:51.050498 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:51.052758 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:51.053599 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:51.053613 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:51.054410 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:51.055197 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:51.055223 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.056336 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.056352 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:51.058051 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:51.058063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.058809 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.058819 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:51.059147 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:51.059846 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:51.059879 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:51.062476 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:51.066452 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:51.066458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:51.066470 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:51.066845 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:51.067245 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:51.067248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:51.067250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:51.067416 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:51.068049 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:51.068054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:51.068331 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 02:59:51.068532 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 02:59:51.068866 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 02:59:51.071506 4180 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38553
I20260926 02:59:51.071799 4180 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:51.072554 4180 heartbeater.cc:511] Master 127.0.97.62:38553 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 02:59:51.074903 3991 ts_manager.cc:194] Registered new tserver with Master: d93e557dc49a4fa89ff20ba23b9ddfb1 (127.0.97.1:39847)
I20260926 02:59:51.076983 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:51.096729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38553
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:51.404744 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:51.405092 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:51.405189 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:51.412045 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:51.412135 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:51.412240 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:51.421452 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:51.422621 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:51.423780 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:51.436005 4192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:51.436048 4194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:51.436088 4191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:51.436970 4186 server_base.cc:1034] running on GCE node
I20260926 02:59:51.437561 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:51.438361 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:51.439543 4186 hybrid_clock.cc:648] HybridClock initialized: now 1790391591439493 us; error 48 us; skew 500 ppm
I20260926 02:59:51.439779 4186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:51 dist-test-slave-fccw krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:51.442972 4186 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:51.444192 4186 webserver.cc:460] Webserver started at http://127.0.97.2:35763/ using document root <none> and password file <none>
I20260926 02:59:51.444777 4186 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:51.444881 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:51.445181 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:51.447688 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "450d38d5347845f08ad1781b3118f90a"
format_stamp: "Formatted at 2026-09-26 02:59:51 on dist-test-slave-fccw"
server_key: "9e1748a663d7ba8e10480c6c737df640"
server_key_iv: "f75a1c1c11c7afc1a4b1ff28f54528f5"
server_key_version: "encryptionkey@0"
I20260926 02:59:51.448380 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "450d38d5347845f08ad1781b3118f90a"
format_stamp: "Formatted at 2026-09-26 02:59:51 on dist-test-slave-fccw"
server_key: "9e1748a663d7ba8e10480c6c737df640"
server_key_iv: "f75a1c1c11c7afc1a4b1ff28f54528f5"
server_key_version: "encryptionkey@0"
I20260926 02:59:51.453078 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 02:59:51.456221 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.457487 4186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:59:51.457607 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "450d38d5347845f08ad1781b3118f90a"
format_stamp: "Formatted at 2026-09-26 02:59:51 on dist-test-slave-fccw"
server_key: "9e1748a663d7ba8e10480c6c737df640"
server_key_iv: "f75a1c1c11c7afc1a4b1ff28f54528f5"
server_key_version: "encryptionkey@0"
I20260926 02:59:51.457762 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:51.470271 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:51.474530 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:51.474747 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:51.475554 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:51.476714 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:51.476783 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.476861 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:51.476907 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.509519 4186 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45259
I20260926 02:59:51.509639 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45259 every 8 connection(s)
I20260926 02:59:51.510751 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:51.514204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 4186
I20260926 02:59:51.514385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:51.514887 388 external_mini_cluster.cc:1428] Setting key b43d628c49fd90a43a6226465957dc6a
Sep 26 02:59:51 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:51.531098 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:51.513342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54043 (local address 127.0.97.62:38553)
0926 02:59:51.513546 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:51.513552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:51.514843 (+ 1291us) server_negotiation.cc:407] Connection header received
0926 02:59:51.517272 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:51.517277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:51.517374 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:51.517489 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:51.518921 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.519623 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.521226 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.521663 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.521792 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:51.525746 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:51.525752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:51.525757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:51.525796 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:51.528173 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:51.528870 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:51.528875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:51.528879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:51.529055 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:51.529473 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:51.529477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:51.529481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:51.529748 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:51.529898 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:51.530536 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 02:59:51.530775 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:59:51.532029 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:51.513905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38553 (local address 127.0.97.2:54043)
0926 02:59:51.514680 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:51.514732 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:51.516947 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:51.517716 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:51.517723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:51.518208 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:51.518763 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:51.518779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.519787 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.519792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:51.521074 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:51.521092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.521897 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.521905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:51.522141 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:51.522870 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:51.522906 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:51.525549 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:51.528312 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:51.528321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:51.528337 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:51.528737 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:51.529170 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:51.529175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:51.529179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:51.529355 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:51.529873 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:51.529880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:51.530207 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 02:59:51.530448 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 02:59:51.530942 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":340,"threads_started":1}
I20260926 02:59:51.533591 4314 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38553
I20260926 02:59:51.533891 4314 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 02:59:51.534564 4314 heartbeater.cc:511] Master 127.0.97.62:38553 requested a full tablet report, sending...
I20260926 02:59:51.536027 3991 ts_manager.cc:194] Registered new tserver with Master: 450d38d5347845f08ad1781b3118f90a (127.0.97.2:45259)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 02:59:51.536821 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:51.558220 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38553
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:51.866293 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:51.866581 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:51.866742 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:51.873550 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:51.873684 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:51.873898 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:51.883061 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:51.884297 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:51.885710 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:51.898095 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:51.898164 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:51.898650 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:51.898885 4319 server_base.cc:1034] running on GCE node
I20260926 02:59:51.899458 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:51.900442 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:51.901638 4319 hybrid_clock.cc:648] HybridClock initialized: now 1790391591901604 us; error 50 us; skew 500 ppm
I20260926 02:59:51.901907 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:51 dist-test-slave-fccw krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:51.904747 4319 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:51.906103 4319 webserver.cc:460] Webserver started at http://127.0.97.3:35053/ using document root <none> and password file <none>
I20260926 02:59:51.906715 4319 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:51.906847 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:51.907155 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:51.909931 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "91df815b219e4b20ad766696cee96f35"
format_stamp: "Formatted at 2026-09-26 02:59:51 on dist-test-slave-fccw"
server_key: "904fbc77df0a749dc97ff98bc37ac47d"
server_key_iv: "09d6ac4e86ac9061d9a983e6ef7e2bae"
server_key_version: "encryptionkey@0"
I20260926 02:59:51.910660 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "91df815b219e4b20ad766696cee96f35"
format_stamp: "Formatted at 2026-09-26 02:59:51 on dist-test-slave-fccw"
server_key: "904fbc77df0a749dc97ff98bc37ac47d"
server_key_iv: "09d6ac4e86ac9061d9a983e6ef7e2bae"
server_key_version: "encryptionkey@0"
I20260926 02:59:51.915432 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 02:59:51.918567 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.919739 4319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 02:59:51.919896 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "91df815b219e4b20ad766696cee96f35"
format_stamp: "Formatted at 2026-09-26 02:59:51 on dist-test-slave-fccw"
server_key: "904fbc77df0a749dc97ff98bc37ac47d"
server_key_iv: "09d6ac4e86ac9061d9a983e6ef7e2bae"
server_key_version: "encryptionkey@0"
I20260926 02:59:51.920078 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:51.949841 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:51.953743 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:51.954018 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:51.955090 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:51.956418 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:51.956513 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.956655 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:51.956748 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:51.988225 4319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33405
I20260926 02:59:51.988276 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33405 every 8 connection(s)
I20260926 02:59:51.989463 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:51.995069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 4319
I20260926 02:59:51.995255 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:51.995687 388 external_mini_cluster.cc:1428] Setting key ba65965df5205eb7e355d3a1e950ee57
Sep 26 02:59:51 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:52.007748 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:51.991837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38861 (local address 127.0.97.62:38553)
0926 02:59:51.992039 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:51.992045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:51.993219 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 02:59:51.994921 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:51.994926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:51.995008 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:51.995125 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:51.996803 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.997760 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.999297 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.999729 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.999833 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:52.003025 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:52.003030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:52.003033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:52.003061 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:52.005378 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.005977 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.005981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.005983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.006080 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.006439 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.006442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.006444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.006694 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:52.006793 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:52.007367 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 02:59:52.007520 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 02:59:52.008633 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:51.992366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38553 (local address 127.0.97.3:38861)
0926 02:59:51.993076 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:51.993119 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:51.994667 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:51.995422 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:51.995430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:51.995959 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:51.996608 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:51.996626 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.997915 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.997919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:51.999154 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:51.999165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:51.999915 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:51.999920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:52.000116 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:52.000796 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:52.000817 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:52.002845 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:52.005500 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:52.005506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:52.005517 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:52.005884 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:52.006163 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:52.006166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:52.006169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:52.006354 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:52.006788 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:52.006793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:52.007065 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 02:59:52.007287 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 02:59:52.007643 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":272,"threads_started":1}
I20260926 02:59:52.009941 4447 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38553
I20260926 02:59:52.010244 4447 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:52.010982 4447 heartbeater.cc:511] Master 127.0.97.62:38553 requested a full tablet report, sending...
I20260926 02:59:52.012280 3991 ts_manager.cc:194] Registered new tserver with Master: 91df815b219e4b20ad766696cee96f35 (127.0.97.3:33405)
I20260926 02:59:52.013125 3991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38861
I20260926 02:59:52.022586 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:52.068509 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:52.054517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.0.97.1:39847)
0926 02:59:52.055219 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:52.055228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:52.055252 (+ 24us) server_negotiation.cc:407] Connection header received
0926 02:59:52.055303 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:52.055309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:52.055522 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:52.055676 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:52.056971 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.057997 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.059160 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.059668 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.059830 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:52.063794 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:52.063805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:52.063818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:52.063851 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:52.066316 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.066848 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.066853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.066855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.066960 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.067268 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.067271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.067272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.067552 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:52.067758 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:52.067961 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 02:59:52.068128 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":312,"threads_started":1}
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:52.080219 4180 heartbeater.cc:503] Master 127.0.97.62:38553 was elected leader, sending a full tablet report...
I20260926 02:59:52.085235 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:52.072331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.0.97.62:38553)
0926 02:59:52.072556 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:52.072560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:52.072662 (+ 102us) server_negotiation.cc:407] Connection header received
0926 02:59:52.072794 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:52.072798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:52.072854 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:52.072928 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:52.074046 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.074814 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.076005 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.076386 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.076513 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:52.079965 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:52.079970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:52.079973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:52.080001 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:52.082816 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.083389 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.083394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.083398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.083513 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.083864 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.083870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.083874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.084160 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:52.084400 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:52.084688 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 02:59:52.084911 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 02:59:52.085705 3991 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60914
I20260926 02:59:52.086566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 4052
I20260926 02:59:52.097363 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:52.087354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.0.97.1:39847)
0926 02:59:52.087486 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:52.087490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:52.087825 (+ 335us) server_negotiation.cc:407] Connection header received
0926 02:59:52.087945 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:52.087948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:52.087995 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:52.088061 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:52.089105 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.089711 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.090817 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.091178 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.091266 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:52.093298 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:52.093302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:52.093305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:52.093330 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:52.095512 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.096028 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.096031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.096033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.096127 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.096407 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.096410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.096412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.096604 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:52.096760 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:52.096961 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 02:59:52.097155 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 02:59:52.097833 4175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:52.177634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 4052
I20260926 02:59:52.192843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 4186
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 02:59:52.207337 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:52.193760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.0.97.2:45259)
0926 02:59:52.194336 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:52.194344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:52.194370 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:59:52.194427 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:52.194433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:52.194648 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:52.194775 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:52.195837 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.196777 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.197959 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.198466 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.198555 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:52.202288 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:52.202298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:52.202311 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:52.202340 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:52.205270 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.205827 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.205833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.205835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.205937 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.206215 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.206218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.206219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.206507 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:52.206713 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:52.206901 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 02:59:52.207066 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":309,"threads_started":1}
I20260926 02:59:52.208524 4309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:52.288641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 4186
I20260926 02:59:52.303761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 4319
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 02:59:52.318595 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:52.304768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.0.97.3:33405)
0926 02:59:52.305404 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:52.305410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:52.305438 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:59:52.305491 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:52.305497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:52.305703 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:52.305858 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:52.306970 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.307900 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.309063 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:52.309590 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:52.309683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:52.313544 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:52.313553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:52.313565 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:52.313593 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:52.316393 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.316918 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.316924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.316926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.317086 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:52.317379 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:52.317382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:52.317384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:52.317669 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:52.317873 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:52.318061 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 02:59:52.318231 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":364,"threads_started":1}
I20260926 02:59:52.319761 4442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:52.400040 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 4319
I20260926 02:59:52.415360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 3959
I20260926 02:59:52.416497 4021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:52.516063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 3959
2026-09-26T02:59:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3670 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[4466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[4466](info): set up 2 sockets
Sep 26 02:59:52 dist-test-slave-fccw krb5kdc[4466](info): commencing operation
krb5kdc: starting...
W20260926 02:59:53.577436 388 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.009s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 02:59:53 dist-test-slave-fccw krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T02:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 02:59:53.757933 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36363
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:54.065769 4479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:54.066145 4479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:54.066309 4479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:54.073167 4479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:54.073304 4479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:54.073426 4479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:54.073526 4479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:54.073614 4479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:54.082983 4479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:54.084196 4479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:54.085471 4479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:54.096374 4487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:54.096411 4485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:54.096335 4484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:54.097419 4479 server_base.cc:1034] running on GCE node
I20260926 02:59:54.098129 4479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:54.099378 4479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:54.100606 4479 hybrid_clock.cc:648] HybridClock initialized: now 1790391594100558 us; error 55 us; skew 500 ppm
I20260926 02:59:54.100926 4479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:54 dist-test-slave-fccw krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:54.108870 4479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 02:59:54.109978 4479 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:54.110100 4479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:54.110416 4479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:54.113276 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "a884537e49f54e0abd280f2a837dad30"
format_stamp: "Formatted at 2026-09-26 02:59:54 on dist-test-slave-fccw"
server_key: "ef6faae0c21649537bab835c23192fb6"
server_key_iv: "3930a6b1551873531f9d92eb880e87b2"
server_key_version: "encryptionkey@0"
I20260926 02:59:54.114068 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "a884537e49f54e0abd280f2a837dad30"
format_stamp: "Formatted at 2026-09-26 02:59:54 on dist-test-slave-fccw"
server_key: "ef6faae0c21649537bab835c23192fb6"
server_key_iv: "3930a6b1551873531f9d92eb880e87b2"
server_key_version: "encryptionkey@0"
I20260926 02:59:54.118814 4479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 02:59:54.121994 4493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:54.123198 4479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 02:59:54.123368 4479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "a884537e49f54e0abd280f2a837dad30"
format_stamp: "Formatted at 2026-09-26 02:59:54 on dist-test-slave-fccw"
server_key: "ef6faae0c21649537bab835c23192fb6"
server_key_iv: "3930a6b1551873531f9d92eb880e87b2"
server_key_version: "encryptionkey@0"
I20260926 02:59:54.123564 4479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:54.155809 4479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:54.160759 4479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:54.161039 4479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:54.178018 4479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36363
I20260926 02:59:54.178059 4544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36363 every 8 connection(s)
I20260926 02:59:54.178849 4479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:54.182367 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:54.188501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 4479
I20260926 02:59:54.188748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:54.189400 388 external_mini_cluster.cc:1428] Setting key c54580cae83c63795181a9760933059c
I20260926 02:59:54.192276 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30: Bootstrap starting.
I20260926 02:59:54.195586 4545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:54.196619 4545 log.cc:824] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:54.199373 4545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30: No bootstrap required, opened a new log
Sep 26 02:59:54 dist-test-slave-fccw krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:54.208387 4545 raft_consensus.cc:348] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a884537e49f54e0abd280f2a837dad30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36363 } }
I20260926 02:59:54.208824 4545 raft_consensus.cc:374] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:54.208918 4545 raft_consensus.cc:729] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a884537e49f54e0abd280f2a837dad30, State: Initialized, Role: FOLLOWER
I20260926 02:59:54.209589 4545 consensus_queue.cc:260] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a884537e49f54e0abd280f2a837dad30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36363 } }
I20260926 02:59:54.209823 4545 raft_consensus.cc:388] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:54.209997 4545 raft_consensus.cc:482] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:54.210167 4545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:54.212142 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:54.191195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.0.97.62:36363)
0926 02:59:54.191930 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:54.191941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:54.191982 (+ 41us) server_negotiation.cc:407] Connection header received
0926 02:59:54.193676 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:54.193696 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:54.194091 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:54.194378 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:54.196159 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:54.197512 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:54.198806 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:54.199804 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:54.199995 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:54.203589 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:54.203598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:54.203616 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:54.203665 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:54.207473 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:54.208169 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:54.208177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:54.208184 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:54.208431 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:54.208849 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:54.208857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:54.208860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:54.209713 (+ 853us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:54.210006 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:54.210463 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 02:59:54.210872 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":335,"threads_started":1}
I20260926 02:59:54.214039 4545 raft_consensus.cc:504] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a884537e49f54e0abd280f2a837dad30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36363 } }
I20260926 02:59:54.214671 4545 leader_election.cc:302] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a884537e49f54e0abd280f2a837dad30; no voters:
I20260926 02:59:54.215261 4545 leader_election.cc:288] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:54.215363 4550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:54.216086 4550 raft_consensus.cc:686] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [term 1 LEADER]: Becoming Leader. State: Replica: a884537e49f54e0abd280f2a837dad30, State: Running, Role: LEADER
I20260926 02:59:54.216697 4550 consensus_queue.cc:237] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a884537e49f54e0abd280f2a837dad30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36363 } }
I20260926 02:59:54.217496 4545 sys_catalog.cc:564] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:54.219533 4552 sys_catalog.cc:455] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a884537e49f54e0abd280f2a837dad30. Latest consensus state: current_term: 1 leader_uuid: "a884537e49f54e0abd280f2a837dad30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a884537e49f54e0abd280f2a837dad30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36363 } } }
I20260926 02:59:54.219760 4551 sys_catalog.cc:455] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a884537e49f54e0abd280f2a837dad30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a884537e49f54e0abd280f2a837dad30" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36363 } } }
I20260926 02:59:54.219980 4552 sys_catalog.cc:458] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:54.220045 4551 sys_catalog.cc:458] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:54.221292 4558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:54.226431 4558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:54.234328 4558 catalog_manager.cc:1347] Generated new cluster ID: 4bf7677700d74529b9552dced49a2634
I20260926 02:59:54.234442 4558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:54.247921 4558 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:54.249171 4558 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:54.260461 4558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30: Generated new TSK 0
I20260926 02:59:54.261263 4558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 02:59:54.312130 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36363
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:54.638432 4571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:54.638695 4571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:54.638827 4571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:54.645974 4571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:54.646103 4571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:54.646298 4571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:54.655957 4571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:54.657217 4571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:54.658548 4571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:54.671350 4579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:54.671342 4576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:54.671342 4577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:54.671993 4571 server_base.cc:1034] running on GCE node
I20260926 02:59:54.672600 4571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:54.673511 4571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:54.674724 4571 hybrid_clock.cc:648] HybridClock initialized: now 1790391594674690 us; error 48 us; skew 500 ppm
I20260926 02:59:54.675025 4571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:54 dist-test-slave-fccw krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:54.678053 4571 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 02:59:54.679030 4571 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:54.679112 4571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:54.679371 4571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:54.682328 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "71811cd10be24aa4bd2aab8d719b7cd5"
format_stamp: "Formatted at 2026-09-26 02:59:54 on dist-test-slave-fccw"
server_key: "28c495d1b7ced6a3214c39488822f36c"
server_key_iv: "80ae555bd674d0e0319917c8fb7f6416"
server_key_version: "encryptionkey@0"
I20260926 02:59:54.683043 4571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "71811cd10be24aa4bd2aab8d719b7cd5"
format_stamp: "Formatted at 2026-09-26 02:59:54 on dist-test-slave-fccw"
server_key: "28c495d1b7ced6a3214c39488822f36c"
server_key_iv: "80ae555bd674d0e0319917c8fb7f6416"
server_key_version: "encryptionkey@0"
I20260926 02:59:54.687623 4571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 02:59:54.690770 4585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:54.692014 4571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 02:59:54.692170 4571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "71811cd10be24aa4bd2aab8d719b7cd5"
format_stamp: "Formatted at 2026-09-26 02:59:54 on dist-test-slave-fccw"
server_key: "28c495d1b7ced6a3214c39488822f36c"
server_key_iv: "80ae555bd674d0e0319917c8fb7f6416"
server_key_version: "encryptionkey@0"
I20260926 02:59:54.692361 4571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:54.729969 4571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:54.734084 4571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:54.734360 4571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:54.735332 4571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:54.736609 4571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:54.736693 4571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:54.736845 4571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:54.736941 4571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:54.768693 4571 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37517
I20260926 02:59:54.768748 4697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37517 every 8 connection(s)
I20260926 02:59:54.769585 4571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:54.771817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 4571
I20260926 02:59:54.772022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:54.772742 388 external_mini_cluster.cc:1428] Setting key 02eebffb9de4fc890b661362a208d946
Sep 26 02:59:54 dist-test-slave-fccw krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:54.789666 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:54.771912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43451 (local address 127.0.97.62:36363)
0926 02:59:54.772382 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:54.772389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:54.773424 (+ 1035us) server_negotiation.cc:407] Connection header received
0926 02:59:54.775174 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:54.775180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:54.775270 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:54.775387 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:54.776800 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:54.777959 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:54.779448 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:54.779919 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:54.780009 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:54.784137 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:54.784142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:54.784145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:54.784175 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:54.786680 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:54.787441 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:54.787448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:54.787452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:54.787571 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:54.787980 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:54.787986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:54.787990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:54.788239 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:54.788398 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:54.789098 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 02:59:54.789319 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 02:59:54.790606 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:54.772473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36363 (local address 127.0.97.1:43451)
0926 02:59:54.773251 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:54.773297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:54.774916 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:54.775562 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:54.775569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:54.776061 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:54.776607 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:54.776624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:54.778100 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:54.778103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:54.779302 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:54.779315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:54.780160 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:54.780168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:54.780497 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:54.781318 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:54.781350 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:54.783931 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:54.786824 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:54.786833 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:54.786851 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:54.787321 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:54.787678 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:54.787684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:54.787688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:54.787869 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:54.788381 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:54.788388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:54.788700 (+ 312us) client_negotiation.cc:769] Sending connection context
0926 02:59:54.788978 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 02:59:54.789503 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":324,"threads_started":1}
I20260926 02:59:54.792212 4698 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36363
I20260926 02:59:54.792567 4698 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 02:59:54.793368 4698 heartbeater.cc:511] Master 127.0.97.62:36363 requested a full tablet report, sending...
I20260926 02:59:54.795552 4510 ts_manager.cc:194] Registered new tserver with Master: 71811cd10be24aa4bd2aab8d719b7cd5 (127.0.97.1:37517)
I20260926 02:59:54.797672 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 02:59:54.816031 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36363
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:55.129350 4704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:55.129638 4704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:55.129815 4704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:55.136755 4704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:55.136909 4704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:55.137104 4704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:55.146275 4704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:55.147485 4704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:55.148705 4704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:55.161242 4710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:55.161195 4709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:55.161957 4712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:55.162132 4704 server_base.cc:1034] running on GCE node
I20260926 02:59:55.162721 4704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:55.163564 4704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:55.164775 4704 hybrid_clock.cc:648] HybridClock initialized: now 1790391595164732 us; error 57 us; skew 500 ppm
I20260926 02:59:55.165117 4704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:55 dist-test-slave-fccw krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:55.167898 4704 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 02:59:55.169077 4704 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:55.169200 4704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:55.169503 4704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:55.172202 4704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "2a783c869e8d42568d8289ba5a060fbd"
format_stamp: "Formatted at 2026-09-26 02:59:55 on dist-test-slave-fccw"
server_key: "5aea046589594862a99cb05a3adc0d49"
server_key_iv: "a0ccfc04203548ee05cafccd1cb296a3"
server_key_version: "encryptionkey@0"
I20260926 02:59:55.172955 4704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2a783c869e8d42568d8289ba5a060fbd"
format_stamp: "Formatted at 2026-09-26 02:59:55 on dist-test-slave-fccw"
server_key: "5aea046589594862a99cb05a3adc0d49"
server_key_iv: "a0ccfc04203548ee05cafccd1cb296a3"
server_key_version: "encryptionkey@0"
I20260926 02:59:55.177898 4704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 02:59:55.181885 4718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:55.183490 4704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 02:59:55.183635 4704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "2a783c869e8d42568d8289ba5a060fbd"
format_stamp: "Formatted at 2026-09-26 02:59:55 on dist-test-slave-fccw"
server_key: "5aea046589594862a99cb05a3adc0d49"
server_key_iv: "a0ccfc04203548ee05cafccd1cb296a3"
server_key_version: "encryptionkey@0"
I20260926 02:59:55.183849 4704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:55.201704 4704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:55.205464 4704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:55.205682 4704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:55.206494 4704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:55.207634 4704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:55.207728 4704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:55.207803 4704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:55.207857 4704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:55.239614 4704 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33507
I20260926 02:59:55.239686 4830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33507 every 8 connection(s)
I20260926 02:59:55.240425 4704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:55.243206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 4704
I20260926 02:59:55.243403 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:55.244063 388 external_mini_cluster.cc:1428] Setting key 70c02e4fa373624883b69a7010f62763
Sep 26 02:59:55 dist-test-slave-fccw krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:55.259645 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.242739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43107 (local address 127.0.97.62:36363)
0926 02:59:55.242910 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:55.242914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:55.244285 (+ 1371us) server_negotiation.cc:407] Connection header received
0926 02:59:55.246262 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:55.246266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:55.246342 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:55.246450 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:55.247829 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.248570 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.250099 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.250519 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.250649 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.254161 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:55.254167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:55.254178 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:55.254208 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:55.256662 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:55.257419 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:55.257429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:55.257432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:55.257544 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:55.258056 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:55.258060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:55.258063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:55.258310 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:55.258455 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:55.259099 (+ 644us) server_negotiation.cc:299] Negotiation successful
0926 02:59:55.259315 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:59:55.260569 4833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.243346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36363 (local address 127.0.97.2:43107)
0926 02:59:55.244086 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:55.244156 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:55.246025 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:55.246653 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:55.246660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:55.247135 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:55.247637 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.247652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.248721 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.248732 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.249963 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.249974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.250742 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.250749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.250950 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.251593 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:55.251617 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:55.253930 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:55.256796 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:55.256806 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:55.256822 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:55.257298 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:55.257668 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:55.257680 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:55.257683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:55.257931 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:55.258430 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:55.258437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:55.258763 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 02:59:55.259005 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 02:59:55.259483 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":314,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 02:59:55.262072 4831 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36363
I20260926 02:59:55.262359 4831 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:55.262984 4831 heartbeater.cc:511] Master 127.0.97.62:36363 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 02:59:55.264447 4510 ts_manager.cc:194] Registered new tserver with Master: 2a783c869e8d42568d8289ba5a060fbd (127.0.97.2:33507)
I20260926 02:59:55.265255 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 02:59:55.284430 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36363
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 02:59:55.593271 4836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:55.593562 4836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:55.593727 4836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:55.600803 4836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:55.600945 4836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:55.601154 4836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:55.610380 4836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36363
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:55.611644 4836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:55.612932 4836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:55.625528 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:55.625560 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:55.625627 4844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:55.626516 4836 server_base.cc:1034] running on GCE node
I20260926 02:59:55.627134 4836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:55.628014 4836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:55.629225 4836 hybrid_clock.cc:648] HybridClock initialized: now 1790391595629159 us; error 50 us; skew 500 ppm
I20260926 02:59:55.629558 4836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 02:59:55 dist-test-slave-fccw krb5kdc[4466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 02:59:55.632596 4836 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 02:59:55.633697 4836 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:55.633783 4836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:55.634035 4836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:55.636682 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "561fd35eb16c40da81fa06a54a1f56cf"
format_stamp: "Formatted at 2026-09-26 02:59:55 on dist-test-slave-fccw"
server_key: "591bf563898ef2060496d57567449326"
server_key_iv: "76e2f58023449b91ab38651b5473085c"
server_key_version: "encryptionkey@0"
I20260926 02:59:55.637477 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "561fd35eb16c40da81fa06a54a1f56cf"
format_stamp: "Formatted at 2026-09-26 02:59:55 on dist-test-slave-fccw"
server_key: "591bf563898ef2060496d57567449326"
server_key_iv: "76e2f58023449b91ab38651b5473085c"
server_key_version: "encryptionkey@0"
I20260926 02:59:55.642099 4836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:59:55.645617 4850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:55.646817 4836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 02:59:55.646966 4836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "561fd35eb16c40da81fa06a54a1f56cf"
format_stamp: "Formatted at 2026-09-26 02:59:55 on dist-test-slave-fccw"
server_key: "591bf563898ef2060496d57567449326"
server_key_iv: "76e2f58023449b91ab38651b5473085c"
server_key_version: "encryptionkey@0"
I20260926 02:59:55.647198 4836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:55.660280 4836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:55.664227 4836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:55.664498 4836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:55.665477 4836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:55.666651 4836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:55.666752 4836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:55.666874 4836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:55.666962 4836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:55.698872 4836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35065
I20260926 02:59:55.698915 4962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35065 every 8 connection(s)
I20260926 02:59:55.699673 4836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:55.700559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 4836
I20260926 02:59:55.700762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:55.701318 388 external_mini_cluster.cc:1428] Setting key 7331df49a3a4d82c2ebcff5f4d6eb90c
Sep 26 02:59:55 dist-test-slave-fccw krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 02:59:55.718046 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.702281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45777 (local address 127.0.97.62:36363)
0926 02:59:55.702477 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:55.702483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:55.703622 (+ 1139us) server_negotiation.cc:407] Connection header received
0926 02:59:55.705567 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:55.705572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:55.705676 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:55.705797 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:55.707199 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.707936 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.709427 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.709778 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.709899 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.712962 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:55.712966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:55.712969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:55.713040 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:55.715458 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:55.716291 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:55.716294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:55.716297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:55.716398 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:55.716726 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:55.716729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:55.716731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:55.716950 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:55.717120 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:55.717690 (+ 570us) server_negotiation.cc:299] Negotiation successful
0926 02:59:55.717831 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:59:55.718883 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.702733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36363 (local address 127.0.97.3:45777)
0926 02:59:55.703485 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:55.703529 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:55.705338 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:55.706028 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:55.706036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:55.706503 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:55.707018 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.707034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.708078 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.708087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.709289 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.709300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.709979 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.709986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.710197 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.710871 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 02:59:55.710895 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:55.712788 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:55.715592 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:55.715598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:55.715627 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:55.716175 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:55.716487 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 02:59:55.716490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 02:59:55.716492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 02:59:55.716639 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 02:59:55.717105 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:55.717110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:55.717398 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 02:59:55.717606 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 02:59:55.717937 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":355,"threads_started":1}
I20260926 02:59:55.720045 4963 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36363
I20260926 02:59:55.720304 4963 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:55.720863 4963 heartbeater.cc:511] Master 127.0.97.62:36363 requested a full tablet report, sending...
I20260926 02:59:55.722281 4510 ts_manager.cc:194] Registered new tserver with Master: 561fd35eb16c40da81fa06a54a1f56cf (127.0.97.3:35065)
I20260926 02:59:55.723030 4510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45777
I20260926 02:59:55.728193 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:59:55.744086 4700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.732317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.0.97.62:36363)
0926 02:59:55.732455 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:55.732459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:55.732909 (+ 450us) server_negotiation.cc:407] Connection header received
0926 02:59:55.733132 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:55.733137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:55.733208 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:55.733307 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:55.734659 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.735351 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.736518 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.736934 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.737107 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.739247 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:55.739253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:55.739258 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:55.739294 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:55.741621 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:55.742264 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:55.742269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:55.742273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:55.742385 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:55.742729 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:55.742735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:55.742738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:55.743020 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:55.743276 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:55.743579 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 02:59:55.743793 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 02:59:55.747646 4510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:59:55.754319 4510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:55.783617 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.765752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33507 (local address 127.0.0.1:44790)
0926 02:59:55.774582 (+ 8830us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:55.774598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:55.774697 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:55.775727 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:55.775732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:55.775755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:55.776132 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.776144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.778294 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.778302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.780949 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.780966 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.782620 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.782626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.782814 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.782880 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 02:59:55.782981 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:59:55.783157 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8727,"mutex_wait_us":26,"thread_start_us":6389,"threads_started":1}
I20260926 02:59:55.784042 4977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.765275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37517 (local address 127.0.0.1:60568)
0926 02:59:55.772496 (+ 7221us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:55.772538 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:55.772702 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:55.775302 (+ 2600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:55.775309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:55.775336 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:55.775671 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.775684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.779131 (+ 3447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.779139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.781457 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.781468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.783193 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.783199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.783371 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.783416 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 02:59:55.783517 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 02:59:55.783645 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6974,"mutex_wait_us":4,"thread_start_us":10619,"threads_started":1}
I20260926 02:59:55.784318 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.766389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44790 (local address 127.0.97.2:33507)
0926 02:59:55.775153 (+ 8764us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:55.775162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:55.775188 (+ 26us) server_negotiation.cc:407] Connection header received
0926 02:59:55.775287 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:55.775294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:55.775504 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:55.775706 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:55.776307 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.778103 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.781181 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.782405 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.783418 (+ 1013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.783607 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:55.783698 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 02:59:55.783852 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8615,"thread_start_us":8505,"threads_started":1}
I20260926 02:59:55.784735 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.765421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.0.97.1:37517)
0926 02:59:55.774428 (+ 9007us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:55.774437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:55.774465 (+ 28us) server_negotiation.cc:407] Connection header received
0926 02:59:55.774563 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:55.774572 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:55.774822 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:55.775031 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:55.775870 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.778937 (+ 3067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.781594 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.782979 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.783090 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.783295 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:55.784272 (+ 977us) server_negotiation.cc:299] Negotiation successful
0926 02:59:55.784432 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8823,"thread_start_us":8677,"threads_started":1}
I20260926 02:59:55.787890 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.776981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35065 (local address 127.0.0.1:49766)
0926 02:59:55.780194 (+ 3213us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:55.780212 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:55.780341 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:55.780881 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:55.780889 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:55.780913 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:55.781400 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.781418 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.784108 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.784113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.785903 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:55.785914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.787122 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.787128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:55.787309 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.787348 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:59:55.787436 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 02:59:55.787546 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3056,"mutex_wait_us":44,"thread_start_us":2897,"threads_started":1}
I20260926 02:59:55.788527 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:55.776589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.0.97.3:35065)
0926 02:59:55.779877 (+ 3288us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:55.779885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:55.780309 (+ 424us) server_negotiation.cc:407] Connection header received
0926 02:59:55.780499 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:55.780509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:55.780693 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:55.780877 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:55.782360 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.783928 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.786064 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:55.786934 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:55.787078 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:55.787293 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:55.787491 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 02:59:55.787634 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3131,"thread_start_us":2978,"threads_started":1}
I20260926 02:59:55.788127 4633 tablet_service.cc:1467] Processing CreateTablet for tablet d5f5f14193a941c69cc6afc74de83111 (DEFAULT_TABLE table=test-table [id=03758ef6316a4b5d89b5bdb4896b4a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:55.788126 4766 tablet_service.cc:1467] Processing CreateTablet for tablet d5f5f14193a941c69cc6afc74de83111 (DEFAULT_TABLE table=test-table [id=03758ef6316a4b5d89b5bdb4896b4a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:55.789582 4633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f5f14193a941c69cc6afc74de83111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:55.789598 4766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f5f14193a941c69cc6afc74de83111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:55.791356 4898 tablet_service.cc:1467] Processing CreateTablet for tablet d5f5f14193a941c69cc6afc74de83111 (DEFAULT_TABLE table=test-table [id=03758ef6316a4b5d89b5bdb4896b4a20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:55.792377 4898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f5f14193a941c69cc6afc74de83111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:55.800463 4698 heartbeater.cc:503] Master 127.0.97.62:36363 was elected leader, sending a full tablet report...
I20260926 02:59:55.805032 4981 tablet_bootstrap.cc:492] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Bootstrap starting.
I20260926 02:59:55.805930 4983 tablet_bootstrap.cc:492] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Bootstrap starting.
I20260926 02:59:55.807142 4982 tablet_bootstrap.cc:492] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Bootstrap starting.
I20260926 02:59:55.808698 4981 tablet_bootstrap.cc:654] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:55.809083 4983 tablet_bootstrap.cc:654] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:55.809984 4983 log.cc:824] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:55.809983 4981 log.cc:824] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:55.810431 4982 tablet_bootstrap.cc:654] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:55.811705 4982 log.cc:824] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:55.812930 4983 tablet_bootstrap.cc:492] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: No bootstrap required, opened a new log
I20260926 02:59:55.813207 4983 ts_tablet_manager.cc:1397] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 02:59:55.813469 4981 tablet_bootstrap.cc:492] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: No bootstrap required, opened a new log
I20260926 02:59:55.813711 4981 ts_tablet_manager.cc:1397] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260926 02:59:55.814553 4982 tablet_bootstrap.cc:492] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: No bootstrap required, opened a new log
I20260926 02:59:55.814797 4982 ts_tablet_manager.cc:1397] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:59:55.820631 4983 raft_consensus.cc:348] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:55.821116 4983 raft_consensus.cc:374] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:55.821228 4983 raft_consensus.cc:729] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 561fd35eb16c40da81fa06a54a1f56cf, State: Initialized, Role: FOLLOWER
I20260926 02:59:55.821841 4983 consensus_queue.cc:260] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:55.822953 4963 heartbeater.cc:503] Master 127.0.97.62:36363 was elected leader, sending a full tablet report...
I20260926 02:59:55.823498 4983 ts_tablet_manager.cc:1428] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260926 02:59:55.824534 4981 raft_consensus.cc:348] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:55.824663 4982 raft_consensus.cc:348] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:55.824954 4981 raft_consensus.cc:374] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:55.825083 4982 raft_consensus.cc:374] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:55.825091 4981 raft_consensus.cc:729] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a783c869e8d42568d8289ba5a060fbd, State: Initialized, Role: FOLLOWER
I20260926 02:59:55.825186 4982 raft_consensus.cc:729] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71811cd10be24aa4bd2aab8d719b7cd5, State: Initialized, Role: FOLLOWER
I20260926 02:59:55.825644 4981 consensus_queue.cc:260] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:55.825779 4982 consensus_queue.cc:260] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:55.827522 4981 ts_tablet_manager.cc:1428] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260926 02:59:55.828003 4831 heartbeater.cc:503] Master 127.0.97.62:36363 was elected leader, sending a full tablet report...
I20260926 02:59:55.829238 4982 ts_tablet_manager.cc:1428] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
W20260926 02:59:55.951097 4964 tablet.cc:2369] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:59:55.992220 4832 tablet.cc:2369] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 02:59:56.021701 4699 tablet.cc:2369] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:59:56.069868 4987 raft_consensus.cc:482] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:56.070114 4987 raft_consensus.cc:504] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:56.071621 4987 leader_election.cc:288] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71811cd10be24aa4bd2aab8d719b7cd5 (127.0.97.1:37517), 2a783c869e8d42568d8289ba5a060fbd (127.0.97.2:33507)
I20260926 02:59:56.077965 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.072051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45551 (local address 127.0.97.1:37517)
0926 02:59:56.072222 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:56.072229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:56.072249 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:59:56.072514 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:56.072518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:56.072576 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:56.072676 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:56.073196 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.074163 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.075940 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.077048 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.077184 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.077262 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:56.077605 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 02:59:56.077703 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:59:56.078019 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.071848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37517 (local address 127.0.97.3:45551)
0926 02:59:56.072168 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:56.072188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:56.072328 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:56.072727 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:56.072732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:56.072757 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:56.072999 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:56.073038 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.074311 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.074317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:56.075790 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:56.075800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.077235 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.077241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:56.077440 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.077477 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 02:59:56.077568 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 02:59:56.077720 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 02:59:56.078665 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.072222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58387 (local address 127.0.97.2:33507)
0926 02:59:56.072427 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:56.072435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:56.072921 (+ 486us) server_negotiation.cc:407] Connection header received
0926 02:59:56.073139 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:56.073146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:56.073242 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:56.073310 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 02:59:56.073871 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.074773 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.076665 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.077703 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.077830 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.077931 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:56.078259 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 02:59:56.078387 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 02:59:56.078685 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.072050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33507 (local address 127.0.97.3:58387)
0926 02:59:56.072801 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:56.072820 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:56.072950 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:56.073353 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:56.073358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:56.073381 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 02:59:56.073704 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:56.073714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.074922 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.074925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:56.076494 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:56.076507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.077908 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.077914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:56.078090 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.078129 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 02:59:56.078214 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 02:59:56.078334 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":3,"thread_start_us":437,"threads_started":1}
I20260926 02:59:56.079344 4653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f5f14193a941c69cc6afc74de83111" candidate_uuid: "561fd35eb16c40da81fa06a54a1f56cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" is_pre_election: true
I20260926 02:59:56.079684 4786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f5f14193a941c69cc6afc74de83111" candidate_uuid: "561fd35eb16c40da81fa06a54a1f56cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a783c869e8d42568d8289ba5a060fbd" is_pre_election: true
I20260926 02:59:56.079819 4653 raft_consensus.cc:2454] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561fd35eb16c40da81fa06a54a1f56cf in term 0.
I20260926 02:59:56.080161 4786 raft_consensus.cc:2454] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 561fd35eb16c40da81fa06a54a1f56cf in term 0.
I20260926 02:59:56.080513 4851 leader_election.cc:302] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561fd35eb16c40da81fa06a54a1f56cf, 71811cd10be24aa4bd2aab8d719b7cd5; no voters:
I20260926 02:59:56.080904 4987 raft_consensus.cc:2781] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:56.081066 4987 raft_consensus.cc:482] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:56.081180 4987 raft_consensus.cc:3037] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:56.083210 4987 raft_consensus.cc:504] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:56.083797 4987 leader_election.cc:288] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [CANDIDATE]: Term 1 election: Requested vote from peers 71811cd10be24aa4bd2aab8d719b7cd5 (127.0.97.1:37517), 2a783c869e8d42568d8289ba5a060fbd (127.0.97.2:33507)
I20260926 02:59:56.084173 4653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f5f14193a941c69cc6afc74de83111" candidate_uuid: "561fd35eb16c40da81fa06a54a1f56cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71811cd10be24aa4bd2aab8d719b7cd5"
I20260926 02:59:56.084236 4786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f5f14193a941c69cc6afc74de83111" candidate_uuid: "561fd35eb16c40da81fa06a54a1f56cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a783c869e8d42568d8289ba5a060fbd"
I20260926 02:59:56.084349 4653 raft_consensus.cc:3037] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:56.084460 4786 raft_consensus.cc:3037] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:56.086524 4653 raft_consensus.cc:2454] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561fd35eb16c40da81fa06a54a1f56cf in term 1.
I20260926 02:59:56.086665 4786 raft_consensus.cc:2454] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 561fd35eb16c40da81fa06a54a1f56cf in term 1.
I20260926 02:59:56.086892 4851 leader_election.cc:302] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 561fd35eb16c40da81fa06a54a1f56cf, 71811cd10be24aa4bd2aab8d719b7cd5; no voters:
I20260926 02:59:56.087158 4987 raft_consensus.cc:2781] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:56.087599 4987 raft_consensus.cc:686] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 1 LEADER]: Becoming Leader. State: Replica: 561fd35eb16c40da81fa06a54a1f56cf, State: Running, Role: LEADER
I20260926 02:59:56.088045 4987 consensus_queue.cc:237] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } }
I20260926 02:59:56.092686 4509 catalog_manager.cc:5495] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 561fd35eb16c40da81fa06a54a1f56cf (127.0.97.3). New cstate: current_term: 1 leader_uuid: "561fd35eb16c40da81fa06a54a1f56cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "561fd35eb16c40da81fa06a54a1f56cf" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35065 } health_report { overall_health: HEALTHY } } }
I20260926 02:59:56.138500 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.133503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.0.97.3:35065)
0926 02:59:56.133682 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:56.133686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:56.133702 (+ 16us) server_negotiation.cc:407] Connection header received
0926 02:59:56.133892 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:56.133896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:56.133966 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:56.134111 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 02:59:56.134454 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.135154 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.136409 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.136782 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.136929 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.137425 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 02:59:56.137900 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 02:59:56.138049 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:56.138161 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 02:59:56.138240 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 02:59:56.146111 4786 raft_consensus.cc:1261] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 1 FOLLOWER]: Refusing update from remote peer 561fd35eb16c40da81fa06a54a1f56cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:56.146112 4653 raft_consensus.cc:1261] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 1 FOLLOWER]: Refusing update from remote peer 561fd35eb16c40da81fa06a54a1f56cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:56.146934 4987 consensus_queue.cc:1035] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a783c869e8d42568d8289ba5a060fbd" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:56.147224 4991 consensus_queue.cc:1035] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "71811cd10be24aa4bd2aab8d719b7cd5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:56.158224 4995 mvcc.cc:205] Tried to move back new op lower bound from 7333443977805266944 to 7333443977580609536. Current Snapshot: MvccSnapshot[applied={T|T < 7333443977805266944}]
I20260926 02:59:56.159276 4994 mvcc.cc:205] Tried to move back new op lower bound from 7333443977805266944 to 7333443977580609536. Current Snapshot: MvccSnapshot[applied={T|T < 7333443977805266944}]
I20260926 02:59:56.163518 4996 mvcc.cc:205] Tried to move back new op lower bound from 7333443977805266944 to 7333443977580609536. Current Snapshot: MvccSnapshot[applied={T|T < 7333443977805266944}]
I20260926 02:59:56.176316 4509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:56.176563 4509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:56.179862 4509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a884537e49f54e0abd280f2a837dad30: Sending DeleteTablet for 3 replicas of tablet d5f5f14193a941c69cc6afc74de83111
I20260926 02:59:56.180918 4633 tablet_service.cc:1514] Processing DeleteTablet for tablet d5f5f14193a941c69cc6afc74de83111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60568
I20260926 02:59:56.181274 4766 tablet_service.cc:1514] Processing DeleteTablet for tablet d5f5f14193a941c69cc6afc74de83111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44790
I20260926 02:59:56.181380 4898 tablet_service.cc:1514] Processing DeleteTablet for tablet d5f5f14193a941c69cc6afc74de83111 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49766
I20260926 02:59:56.183490 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 4571
I20260926 02:59:56.192417 5004 tablet_replica.cc:330] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: stopping tablet replica
I20260926 02:59:56.193506 5004 raft_consensus.cc:2229] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:56.194684 5004 raft_consensus.cc:2258] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:56.198056 5004 ts_tablet_manager.cc:1905] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:56.207990 5004 ts_tablet_manager.cc:1918] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:56.209394 5004 log.cc:1196] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/d5f5f14193a941c69cc6afc74de83111
I20260926 02:59:56.210021 5004 ts_tablet_manager.cc:1939] T d5f5f14193a941c69cc6afc74de83111 P 71811cd10be24aa4bd2aab8d719b7cd5: Deleting consensus metadata
I20260926 02:59:56.211784 4494 catalog_manager.cc:4841] TS 71811cd10be24aa4bd2aab8d719b7cd5 (127.0.97.1:37517): tablet d5f5f14193a941c69cc6afc74de83111 (table test-table [id=03758ef6316a4b5d89b5bdb4896b4a20]) successfully deleted
I20260926 02:59:56.212076 5006 tablet_replica.cc:330] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: stopping tablet replica
I20260926 02:59:56.212504 5006 raft_consensus.cc:2229] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:56.213294 5006 raft_consensus.cc:2258] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:56.213722 5005 tablet_replica.cc:330] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: stopping tablet replica
I20260926 02:59:56.214195 5005 raft_consensus.cc:2229] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:56.214712 5005 raft_consensus.cc:2258] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:56.215175 5006 ts_tablet_manager.cc:1905] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:56.216779 5005 ts_tablet_manager.cc:1905] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 02:59:56 dist-test-slave-fccw krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 02:59:56.223788 5005 ts_tablet_manager.cc:1918] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:56.224018 5005 log.cc:1196] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/d5f5f14193a941c69cc6afc74de83111
I20260926 02:59:56.224082 5006 ts_tablet_manager.cc:1918] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:56.224354 5006 log.cc:1196] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/d5f5f14193a941c69cc6afc74de83111
I20260926 02:59:56.224623 5005 ts_tablet_manager.cc:1939] T d5f5f14193a941c69cc6afc74de83111 P 2a783c869e8d42568d8289ba5a060fbd: Deleting consensus metadata
I20260926 02:59:56.224984 5006 ts_tablet_manager.cc:1939] T d5f5f14193a941c69cc6afc74de83111 P 561fd35eb16c40da81fa06a54a1f56cf: Deleting consensus metadata
I20260926 02:59:56.226642 4494 catalog_manager.cc:4841] TS 561fd35eb16c40da81fa06a54a1f56cf (127.0.97.3:35065): tablet d5f5f14193a941c69cc6afc74de83111 (table test-table [id=03758ef6316a4b5d89b5bdb4896b4a20]) successfully deleted
I20260926 02:59:56.226728 4495 catalog_manager.cc:4841] TS 2a783c869e8d42568d8289ba5a060fbd (127.0.97.2:33507): tablet d5f5f14193a941c69cc6afc74de83111 (table test-table [id=03758ef6316a4b5d89b5bdb4896b4a20]) successfully deleted
I20260926 02:59:56.228637 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.185303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.97.1:37517)
0926 02:59:56.185480 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:56.185488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:56.209044 (+ 23556us) server_negotiation.cc:407] Connection header received
0926 02:59:56.209553 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:56.209563 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:56.209671 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:56.209777 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:56.212073 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.213393 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.215734 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.216360 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.216470 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.221755 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:56.221768 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:56.221785 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:56.221826 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:56.225484 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:56.226297 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:56.226305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:56.226310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:56.226463 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:56.227006 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:56.227011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:56.227015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:56.227436 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:56.227747 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:56.228061 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 02:59:56.228266 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 02:59:56.229338 4693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:56.330585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 4571
I20260926 02:59:56.353174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 4704
Sep 26 02:59:56 dist-test-slave-fccw krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 02:59:56.369330 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.354018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.0.97.2:33507)
0926 02:59:56.354207 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:56.354214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:56.354235 (+ 21us) server_negotiation.cc:407] Connection header received
0926 02:59:56.354303 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:56.354309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:56.354395 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:56.354507 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:56.355670 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.356778 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.358143 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.358786 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.358908 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.362971 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:56.362992 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:56.363013 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:56.363061 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:56.366709 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:56.367308 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:56.367318 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:56.367322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:56.367487 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:56.367831 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:56.367836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:56.367839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:56.368265 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:56.368553 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:56.368733 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 02:59:56.368939 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 02:59:56.370033 4826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:56.457142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 4704
I20260926 02:59:56.481118 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 4836
Sep 26 02:59:56 dist-test-slave-fccw krb5kdc[4466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 02:59:56.497922 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:56.481852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.97.3:35065)
0926 02:59:56.482015 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:56.482019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:56.482120 (+ 101us) server_negotiation.cc:407] Connection header received
0926 02:59:56.482303 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:56.482307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:56.482367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:56.482448 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:56.484118 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.484928 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.486787 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:56.487212 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:56.487310 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:56.492109 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:56.492120 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:56.492136 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 02:59:56.492171 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:56.495175 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:56.495983 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:56.495990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:56.495993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:56.496137 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 02:59:56.496495 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 02:59:56.496501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 02:59:56.496504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 02:59:56.496829 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:56.497185 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:56.497372 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 02:59:56.497574 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:59:56.498553 4958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:56.617831 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 4836
I20260926 02:59:56.636905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 4479
I20260926 02:59:56.638028 4540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:56.743633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 4479
2026-09-26T02:59:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4231 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T02:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:56Z Disabled control of system clock
I20260926 02:59:56.807199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42755
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:59:57.119127 5015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:57.119377 5015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:57.119459 5015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:57.126463 5015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:57.126560 5015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:57.126595 5015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:57.126645 5015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:57.126688 5015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 02:59:57.135991 5015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42755
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:57.137203 5015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:57.138913 5015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:57.150099 5023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:57.150159 5020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:57.150187 5021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:57.151052 5015 server_base.cc:1034] running on GCE node
I20260926 02:59:57.151857 5015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:57.153277 5015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:57.154500 5015 hybrid_clock.cc:648] HybridClock initialized: now 1790391597154465 us; error 50 us; skew 500 ppm
I20260926 02:59:57.154836 5015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:59:57.156932 5015 webserver.cc:460] Webserver started at http://127.0.97.62:41245/ using document root <none> and password file <none>
I20260926 02:59:57.157609 5015 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:57.157734 5015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:57.158057 5015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:57.160856 5015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "8cc9ef989b23476dba86fe6b4c852099"
format_stamp: "Formatted at 2026-09-26 02:59:57 on dist-test-slave-fccw"
server_key: "bdb517f8564960a9a2e9a1c628a4e939"
server_key_iv: "dddc9645c82a81a1ce2fd94c38ec9322"
server_key_version: "encryptionkey@0"
I20260926 02:59:57.161662 5015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "8cc9ef989b23476dba86fe6b4c852099"
format_stamp: "Formatted at 2026-09-26 02:59:57 on dist-test-slave-fccw"
server_key: "bdb517f8564960a9a2e9a1c628a4e939"
server_key_iv: "dddc9645c82a81a1ce2fd94c38ec9322"
server_key_version: "encryptionkey@0"
I20260926 02:59:57.166370 5015 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 02:59:57.169685 5029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:57.170862 5015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 02:59:57.171030 5015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "8cc9ef989b23476dba86fe6b4c852099"
format_stamp: "Formatted at 2026-09-26 02:59:57 on dist-test-slave-fccw"
server_key: "bdb517f8564960a9a2e9a1c628a4e939"
server_key_iv: "dddc9645c82a81a1ce2fd94c38ec9322"
server_key_version: "encryptionkey@0"
I20260926 02:59:57.171242 5015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:57.176481 5015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:57.177251 5015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:57.177533 5015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:57.194530 5015 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42755
I20260926 02:59:57.194562 5080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42755 every 8 connection(s)
I20260926 02:59:57.195770 5015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:57.199208 5081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:57.203299 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5015
I20260926 02:59:57.203492 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:57.204072 388 external_mini_cluster.cc:1428] Setting key 979f3dd27c634a8388c38bec028ec313
I20260926 02:59:57.209348 5081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099: Bootstrap starting.
I20260926 02:59:57.212054 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:57.205952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.0.97.62:42755)
0926 02:59:57.206935 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:57.206948 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:57.206992 (+ 44us) server_negotiation.cc:407] Connection header received
0926 02:59:57.208567 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:57.208590 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:57.208979 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:57.209416 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:57.210493 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:57.210498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:57.210511 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:57.210524 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:57.210581 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:57.210596 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:57.210762 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:57.210878 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 02:59:57.211141 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":379,"threads_started":1}
I20260926 02:59:57.212859 5081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:57.213892 5081 log.cc:824] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:57.216441 5081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099: No bootstrap required, opened a new log
I20260926 02:59:57.223659 5081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc9ef989b23476dba86fe6b4c852099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260926 02:59:57.224018 5081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:57.224119 5081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc9ef989b23476dba86fe6b4c852099, State: Initialized, Role: FOLLOWER
I20260926 02:59:57.224646 5081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc9ef989b23476dba86fe6b4c852099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260926 02:59:57.224900 5081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:57.225054 5081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:57.225220 5081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:57.227326 5081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc9ef989b23476dba86fe6b4c852099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260926 02:59:57.227789 5081 leader_election.cc:302] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cc9ef989b23476dba86fe6b4c852099; no voters:
I20260926 02:59:57.228312 5081 leader_election.cc:288] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:57.228434 5086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:57.229142 5086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [term 1 LEADER]: Becoming Leader. State: Replica: 8cc9ef989b23476dba86fe6b4c852099, State: Running, Role: LEADER
I20260926 02:59:57.229776 5086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc9ef989b23476dba86fe6b4c852099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } }
I20260926 02:59:57.230429 5081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:57.232697 5088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cc9ef989b23476dba86fe6b4c852099. Latest consensus state: current_term: 1 leader_uuid: "8cc9ef989b23476dba86fe6b4c852099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc9ef989b23476dba86fe6b4c852099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } } }
I20260926 02:59:57.232733 5087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cc9ef989b23476dba86fe6b4c852099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cc9ef989b23476dba86fe6b4c852099" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42755 } } }
I20260926 02:59:57.233111 5087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:57.233107 5088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:57.233907 5095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:57.239117 5095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:57.246733 5095 catalog_manager.cc:1347] Generated new cluster ID: 715f01ba7da140a295848a95521b35dd
I20260926 02:59:57.246825 5095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:57.280656 5095 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:57.281131 5095 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:57.285898 5095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099: Generated new TSK 0
I20260926 02:59:57.286649 5095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 02:59:57.294862 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--builtin_ntp_servers=127.0.97.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:59:57.610623 5105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:57.610878 5105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:57.611011 5105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:57.618018 5105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:57.618105 5105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:57.618296 5105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 02:59:57.627003 5105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:57.628248 5105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:57.629920 5105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:57.642750 5110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:57.642711 5111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:57.642863 5113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:57.643373 5105 server_base.cc:1034] running on GCE node
I20260926 02:59:57.643942 5105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:57.644812 5105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:57.646027 5105 hybrid_clock.cc:648] HybridClock initialized: now 1790391597645993 us; error 49 us; skew 500 ppm
I20260926 02:59:57.646328 5105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:59:57.648419 5105 webserver.cc:460] Webserver started at http://127.0.97.1:46335/ using document root <none> and password file <none>
I20260926 02:59:57.649111 5105 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:57.649236 5105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:57.649539 5105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:57.652182 5105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "1f6299eb936744c0a995b71bbb4dcf4e"
format_stamp: "Formatted at 2026-09-26 02:59:57 on dist-test-slave-fccw"
server_key: "2b64c2ba9179d70a586ba82ebd628012"
server_key_iv: "08bd82be6f58ac276546e7db6fc0b283"
server_key_version: "encryptionkey@0"
I20260926 02:59:57.652988 5105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1f6299eb936744c0a995b71bbb4dcf4e"
format_stamp: "Formatted at 2026-09-26 02:59:57 on dist-test-slave-fccw"
server_key: "2b64c2ba9179d70a586ba82ebd628012"
server_key_iv: "08bd82be6f58ac276546e7db6fc0b283"
server_key_version: "encryptionkey@0"
I20260926 02:59:57.657970 5105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 02:59:57.661190 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:57.662353 5105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 02:59:57.662525 5105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "1f6299eb936744c0a995b71bbb4dcf4e"
format_stamp: "Formatted at 2026-09-26 02:59:57 on dist-test-slave-fccw"
server_key: "2b64c2ba9179d70a586ba82ebd628012"
server_key_iv: "08bd82be6f58ac276546e7db6fc0b283"
server_key_version: "encryptionkey@0"
I20260926 02:59:57.662751 5105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:57.668120 5105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:57.668843 5105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:57.669162 5105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:57.670055 5105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:57.671332 5105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:57.671442 5105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:57.671566 5105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:57.671677 5105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:57.703186 5105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41505
I20260926 02:59:57.703238 5231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41505 every 8 connection(s)
I20260926 02:59:57.704461 5105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 02:59:57.712080 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:57.706804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37345 (local address 127.0.97.62:42755)
0926 02:59:57.706991 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:57.706997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:57.708406 (+ 1409us) server_negotiation.cc:407] Connection header received
0926 02:59:57.710088 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:57.710092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:57.710144 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:57.710239 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:57.711520 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:57.711524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:57.711527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:57.711541 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:57.711567 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:57.711578 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:57.711674 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:57.711833 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 02:59:57.711890 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 02:59:57.712896 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:57.707321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.1:37345)
0926 02:59:57.708256 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:57.708293 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:57.709858 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:57.710444 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:57.710452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:57.710546 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:57.711244 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:57.711246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:57.711283 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:57.711360 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:57.711665 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:57.711672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:57.711677 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:59:57.711874 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 02:59:57.712068 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":284,"threads_started":1}
I20260926 02:59:57.714187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5105
I20260926 02:59:57.714233 5232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42755
I20260926 02:59:57.714382 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 02:59:57.714516 5232 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:57.715174 388 external_mini_cluster.cc:1428] Setting key 014ee890bb53fd20724182049748aa38
I20260926 02:59:57.715153 5232 heartbeater.cc:511] Master 127.0.97.62:42755 requested a full tablet report, sending...
I20260926 02:59:57.717373 5045 ts_manager.cc:194] Registered new tserver with Master: 1f6299eb936744c0a995b71bbb4dcf4e (127.0.97.1:41505)
I20260926 02:59:57.717891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--builtin_ntp_servers=127.0.97.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 02:59:58.025971 5235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:58.026227 5235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:58.026388 5235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:58.033610 5235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:58.033702 5235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:58.033926 5235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 02:59:58.043298 5235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:58.044517 5235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:58.046295 5235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:58.058876 5240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:58.058884 5241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:58.058879 5243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:58.062340 5235 server_base.cc:1034] running on GCE node
I20260926 02:59:58.062901 5235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:58.063766 5235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:58.064985 5235 hybrid_clock.cc:648] HybridClock initialized: now 1790391598064949 us; error 32 us; skew 500 ppm
I20260926 02:59:58.065335 5235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:59:58.067557 5235 webserver.cc:460] Webserver started at http://127.0.97.2:46103/ using document root <none> and password file <none>
I20260926 02:59:58.068209 5235 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:58.068331 5235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:58.068625 5235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:58.071362 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "a2bfe738289746c9b3644b8813f0a5f0"
format_stamp: "Formatted at 2026-09-26 02:59:58 on dist-test-slave-fccw"
server_key: "46975397a9035a78a3b491b5f82f4254"
server_key_iv: "a565a3c4fed60f29e9ab160fbedaf42f"
server_key_version: "encryptionkey@0"
I20260926 02:59:58.072109 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a2bfe738289746c9b3644b8813f0a5f0"
format_stamp: "Formatted at 2026-09-26 02:59:58 on dist-test-slave-fccw"
server_key: "46975397a9035a78a3b491b5f82f4254"
server_key_iv: "a565a3c4fed60f29e9ab160fbedaf42f"
server_key_version: "encryptionkey@0"
I20260926 02:59:58.076944 5235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 02:59:58.080076 5249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:58.081338 5235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 02:59:58.081506 5235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "a2bfe738289746c9b3644b8813f0a5f0"
format_stamp: "Formatted at 2026-09-26 02:59:58 on dist-test-slave-fccw"
server_key: "46975397a9035a78a3b491b5f82f4254"
server_key_iv: "a565a3c4fed60f29e9ab160fbedaf42f"
server_key_version: "encryptionkey@0"
I20260926 02:59:58.081730 5235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:58.087095 5235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:58.087770 5235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:58.088037 5235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:58.088855 5235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:58.090385 5235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:58.090499 5235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:58.090646 5235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:58.090749 5235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:58.122063 5235 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43651
I20260926 02:59:58.122112 5361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43651 every 8 connection(s)
I20260926 02:59:58.123327 5235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 02:59:58.125057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5235
I20260926 02:59:58.125242 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 02:59:58.125780 388 external_mini_cluster.cc:1428] Setting key 6cbd79bd83297052899ebb9fd205687e
I20260926 02:59:58.128635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--builtin_ntp_servers=127.0.97.20:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 02:59:58.131705 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.125942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59993 (local address 127.0.97.62:42755)
0926 02:59:58.126104 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.126108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.127578 (+ 1470us) server_negotiation.cc:407] Connection header received
0926 02:59:58.129362 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.129367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.129422 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.129557 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.130966 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.130971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.130975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.130991 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.131031 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.131042 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.131154 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.131361 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.131423 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 02:59:58.132721 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.126547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.2:59993)
0926 02:59:58.127446 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.127478 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:58.129126 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:58.129787 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:58.129794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:58.129882 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:58.130604 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:58.130607 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:58.130642 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:58.130824 (+ 182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:58.131129 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:58.131139 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:58.131146 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 02:59:58.131438 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 02:59:58.131669 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":326,"threads_started":1}
I20260926 02:59:58.134167 5362 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42755
I20260926 02:59:58.134475 5362 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:58.135001 5362 heartbeater.cc:511] Master 127.0.97.62:42755 requested a full tablet report, sending...
I20260926 02:59:58.136281 5045 ts_manager.cc:194] Registered new tserver with Master: a2bfe738289746c9b3644b8813f0a5f0 (127.0.97.2:43651)
W20260926 02:59:58.453529 5365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:58.453833 5365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:58.453994 5365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:58.461113 5365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:58.461241 5365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:58.461421 5365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 02:59:58.470476 5365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:58.471758 5365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:58.473479 5365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:58.486383 5371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:58.486409 5370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:58.486367 5373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:58.487126 5365 server_base.cc:1034] running on GCE node
I20260926 02:59:58.487749 5365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:58.488549 5365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:58.489766 5365 hybrid_clock.cc:648] HybridClock initialized: now 1790391598489731 us; error 49 us; skew 500 ppm
I20260926 02:59:58.490092 5365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:59:58.492262 5365 webserver.cc:460] Webserver started at http://127.0.97.3:41815/ using document root <none> and password file <none>
I20260926 02:59:58.492905 5365 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:58.493079 5365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:58.493402 5365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:58.496167 5365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "015a206faa844711a653aff5c570aa86"
format_stamp: "Formatted at 2026-09-26 02:59:58 on dist-test-slave-fccw"
server_key: "662251a4af88006733d0235fcbfbd2b2"
server_key_iv: "90639df917e90eb83de892c04a5aff1c"
server_key_version: "encryptionkey@0"
I20260926 02:59:58.496930 5365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "015a206faa844711a653aff5c570aa86"
format_stamp: "Formatted at 2026-09-26 02:59:58 on dist-test-slave-fccw"
server_key: "662251a4af88006733d0235fcbfbd2b2"
server_key_iv: "90639df917e90eb83de892c04a5aff1c"
server_key_version: "encryptionkey@0"
I20260926 02:59:58.501817 5365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 02:59:58.504897 5379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:58.506171 5365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 02:59:58.506347 5365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "015a206faa844711a653aff5c570aa86"
format_stamp: "Formatted at 2026-09-26 02:59:58 on dist-test-slave-fccw"
server_key: "662251a4af88006733d0235fcbfbd2b2"
server_key_iv: "90639df917e90eb83de892c04a5aff1c"
server_key_version: "encryptionkey@0"
I20260926 02:59:58.506563 5365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:58.511816 5365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:58.512543 5365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:58.512806 5365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:58.513675 5365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 02:59:58.514961 5365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 02:59:58.515089 5365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:58.515215 5365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 02:59:58.515302 5365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:58.546883 5365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46789
I20260926 02:59:58.547034 5491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46789 every 8 connection(s)
I20260926 02:59:58.548239 5365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 02:59:58.555896 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.550710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36965 (local address 127.0.97.62:42755)
0926 02:59:58.550889 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.550893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.552301 (+ 1408us) server_negotiation.cc:407] Connection header received
0926 02:59:58.553939 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.553943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.553999 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.554091 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.555334 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.555338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.555348 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.555362 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.555391 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.555402 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.555505 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.555660 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.555707 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 02:59:58.556771 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.551271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42755 (local address 127.0.97.3:36965)
0926 02:59:58.552143 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.552179 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:58.553701 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:58.554291 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:58.554303 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:58.554388 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:58.555060 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:58.555062 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:58.555097 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:58.555176 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:58.555488 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:58.555495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:58.555500 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:59:58.555698 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 02:59:58.555876 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":269,"threads_started":1}
I20260926 02:59:58.557372 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5365
I20260926 02:59:58.557564 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 02:59:58.558022 388 external_mini_cluster.cc:1428] Setting key 4c087b8e85a22a4d19fa0975e1d1f898
I20260926 02:59:58.558262 5492 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42755
I20260926 02:59:58.558565 5492 heartbeater.cc:464] Registering TS with master...
I20260926 02:59:58.559233 5492 heartbeater.cc:511] Master 127.0.97.62:42755 requested a full tablet report, sending...
I20260926 02:59:58.560770 5046 ts_manager.cc:194] Registered new tserver with Master: 015a206faa844711a653aff5c570aa86 (127.0.97.3:46789)
I20260926 02:59:58.573076 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 02:59:58.579487 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.577150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.0.97.62:42755)
0926 02:59:58.577351 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.577360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.577798 (+ 438us) server_negotiation.cc:407] Connection header received
0926 02:59:58.577919 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.577922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.577966 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.578042 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.578925 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.578929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.578932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.578943 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.578967 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.578977 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.579063 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.579147 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.579191 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 02:59:58.583244 5046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 02:59:58.589815 5046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 02:59:58.605207 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.600667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43651 (local address 127.0.0.1:53660)
0926 02:59:58.601477 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.601530 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:58.601709 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:58.602837 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:58.602844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:58.602931 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:58.603732 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:58.603738 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:58.603770 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:58.603845 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:58.604310 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:58.604319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:58.604327 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 02:59:58.604531 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 02:59:58.604632 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":281,"threads_started":1}
I20260926 02:59:58.605921 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.600962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.0.97.2:43651)
0926 02:59:58.602211 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.602217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.602240 (+ 23us) server_negotiation.cc:407] Connection header received
0926 02:59:58.602331 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.602342 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.602605 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.602826 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.604129 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.604135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.604145 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.604157 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.604190 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.604205 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.604376 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.605465 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.605561 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":957,"threads_started":1}
I20260926 02:59:58.609236 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.601436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46789 (local address 127.0.0.1:40698)
0926 02:59:58.604812 (+ 3376us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.604835 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:58.604933 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:58.605905 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:58.605911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:58.605934 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:58.607250 (+ 1316us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:58.607254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:58.607269 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:58.607322 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:58.608589 (+ 1267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:58.608593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:58.608597 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:59:58.608748 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 02:59:58.608798 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3216,"thread_start_us":6094,"threads_started":1}
I20260926 02:59:58.609885 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.601982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.0.97.3:46789)
0926 02:59:58.602696 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.602706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.605064 (+ 2358us) server_negotiation.cc:407] Connection header received
0926 02:59:58.605143 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.605153 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.605339 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.605458 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.607787 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.608250 (+ 463us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.608266 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.608286 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.608347 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.608371 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.608535 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.609429 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.609517 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":391,"threads_started":1}
I20260926 02:59:58.610944 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.601766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34058)
0926 02:59:58.605929 (+ 4163us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.605950 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:58.606047 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:58.608664 (+ 2617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:58.608668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:58.608689 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:58.609242 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:58.609244 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:58.609255 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:58.609313 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:58.610454 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:58.610458 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:58.610463 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:59:58.610568 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 02:59:58.610615 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4047,"thread_start_us":2765,"threads_started":1}
I20260926 02:59:58.610456 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9dcda72df1410c961756aee4c58387 (DEFAULT_TABLE table=test-table [id=87205565445448eead3965d8ddc660e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:58.611059 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.602262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.97.1:41505)
0926 02:59:58.608164 (+ 5902us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.608169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.608189 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:59:58.608270 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.608276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.608457 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.608965 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.610279 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.610284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.610292 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.610309 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.610341 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.610353 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.610491 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.610677 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.610761 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5761,"thread_start_us":5653,"threads_started":1}
I20260926 02:59:58.612294 5297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9dcda72df1410c961756aee4c58387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:58.613276 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9dcda72df1410c961756aee4c58387 (DEFAULT_TABLE table=test-table [id=87205565445448eead3965d8ddc660e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:58.614638 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9dcda72df1410c961756aee4c58387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:58.614874 5167 tablet_service.cc:1467] Processing CreateTablet for tablet 3e9dcda72df1410c961756aee4c58387 (DEFAULT_TABLE table=test-table [id=87205565445448eead3965d8ddc660e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 02:59:58.615986 5167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e9dcda72df1410c961756aee4c58387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:58.627312 5510 tablet_bootstrap.cc:492] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: Bootstrap starting.
I20260926 02:59:58.628502 5512 tablet_bootstrap.cc:492] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Bootstrap starting.
I20260926 02:59:58.630221 5511 tablet_bootstrap.cc:492] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Bootstrap starting.
I20260926 02:59:58.630753 5510 tablet_bootstrap.cc:654] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:58.631973 5512 tablet_bootstrap.cc:654] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:58.631965 5510 log.cc:824] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:58.633216 5512 log.cc:824] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:58.633376 5511 tablet_bootstrap.cc:654] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:58.634605 5511 log.cc:824] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:58.635475 5510 tablet_bootstrap.cc:492] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: No bootstrap required, opened a new log
I20260926 02:59:58.635759 5510 ts_tablet_manager.cc:1397] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260926 02:59:58.636230 5512 tablet_bootstrap.cc:492] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: No bootstrap required, opened a new log
I20260926 02:59:58.636476 5512 ts_tablet_manager.cc:1397] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 02:59:58.637279 5511 tablet_bootstrap.cc:492] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: No bootstrap required, opened a new log
I20260926 02:59:58.637512 5511 ts_tablet_manager.cc:1397] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 02:59:58.643684 5512 raft_consensus.cc:348] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.644023 5512 raft_consensus.cc:374] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:58.644124 5512 raft_consensus.cc:729] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015a206faa844711a653aff5c570aa86, State: Initialized, Role: FOLLOWER
I20260926 02:59:58.644722 5512 consensus_queue.cc:260] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.645790 5492 heartbeater.cc:503] Master 127.0.97.62:42755 was elected leader, sending a full tablet report...
I20260926 02:59:58.646286 5512 ts_tablet_manager.cc:1428] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260926 02:59:58.646209 5510 raft_consensus.cc:348] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.646603 5510 raft_consensus.cc:374] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:58.646695 5510 raft_consensus.cc:729] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6299eb936744c0a995b71bbb4dcf4e, State: Initialized, Role: FOLLOWER
I20260926 02:59:58.647236 5510 consensus_queue.cc:260] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.648345 5511 raft_consensus.cc:348] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.648669 5511 raft_consensus.cc:374] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:58.648761 5510 ts_tablet_manager.cc:1428] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 02:59:58.648787 5511 raft_consensus.cc:729] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2bfe738289746c9b3644b8813f0a5f0, State: Initialized, Role: FOLLOWER
I20260926 02:59:58.649310 5511 consensus_queue.cc:260] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.649674 5232 heartbeater.cc:503] Master 127.0.97.62:42755 was elected leader, sending a full tablet report...
I20260926 02:59:58.650314 5362 heartbeater.cc:503] Master 127.0.97.62:42755 was elected leader, sending a full tablet report...
I20260926 02:59:58.650820 5511 ts_tablet_manager.cc:1428] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260926 02:59:58.706629 5233 tablet.cc:2369] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 02:59:58.718284 5516 raft_consensus.cc:482] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 02:59:58.718580 5516 raft_consensus.cc:504] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.720184 5516 leader_election.cc:288] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2bfe738289746c9b3644b8813f0a5f0 (127.0.97.2:43651), 1f6299eb936744c0a995b71bbb4dcf4e (127.0.97.1:41505)
I20260926 02:59:58.722558 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.720470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43651 (local address 127.0.97.3:38983)
0926 02:59:58.720639 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.720655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:58.720739 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:58.721111 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:58.721115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:58.721141 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:58.721767 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:58.721772 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:58.721785 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:58.721827 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:58.722159 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:58.722164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:58.722168 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 02:59:58.722274 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 02:59:58.722328 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 02:59:58.722771 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.720477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38983 (local address 127.0.97.2:43651)
0926 02:59:58.720689 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.720694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.720716 (+ 22us) server_negotiation.cc:407] Connection header received
0926 02:59:58.720858 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.720865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.720924 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.721067 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.721981 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.721988 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.721992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.722009 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.722047 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.722064 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.722184 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.722311 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.722465 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:59:58.723526 5317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9dcda72df1410c961756aee4c58387" candidate_uuid: "015a206faa844711a653aff5c570aa86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bfe738289746c9b3644b8813f0a5f0" is_pre_election: true
I20260926 02:59:58.723981 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.721177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47061 (local address 127.0.97.1:41505)
0926 02:59:58.721377 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.721384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.722017 (+ 633us) server_negotiation.cc:407] Connection header received
0926 02:59:58.722168 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.722173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.722242 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.722329 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.723372 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.723377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.723382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.723399 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.723435 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.723447 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.723557 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.723643 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.723714 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 02:59:58.724012 5317 raft_consensus.cc:2454] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 015a206faa844711a653aff5c570aa86 in term 0.
I20260926 02:59:58.724143 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.720892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.97.3:47061)
0926 02:59:58.721914 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.721931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:58.722033 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:58.722420 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:58.722425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:58.722456 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0926 02:59:58.723202 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 02:59:58.723206 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 02:59:58.723220 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 02:59:58.723257 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 02:59:58.723546 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 02:59:58.723550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 02:59:58.723555 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 02:59:58.723658 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 02:59:58.723711 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"thread_start_us":465,"threads_started":1}
I20260926 02:59:58.724637 5380 leader_election.cc:302] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015a206faa844711a653aff5c570aa86, a2bfe738289746c9b3644b8813f0a5f0; no voters:
I20260926 02:59:58.725066 5516 raft_consensus.cc:2781] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 02:59:58.725196 5516 raft_consensus.cc:482] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 02:59:58.725044 5187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9dcda72df1410c961756aee4c58387" candidate_uuid: "015a206faa844711a653aff5c570aa86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" is_pre_election: true
I20260926 02:59:58.725309 5516 raft_consensus.cc:3037] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:58.725538 5187 raft_consensus.cc:2454] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 015a206faa844711a653aff5c570aa86 in term 0.
I20260926 02:59:58.727634 5516 raft_consensus.cc:504] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.728221 5516 leader_election.cc:288] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [CANDIDATE]: Term 1 election: Requested vote from peers a2bfe738289746c9b3644b8813f0a5f0 (127.0.97.2:43651), 1f6299eb936744c0a995b71bbb4dcf4e (127.0.97.1:41505)
I20260926 02:59:58.728534 5317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9dcda72df1410c961756aee4c58387" candidate_uuid: "015a206faa844711a653aff5c570aa86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2bfe738289746c9b3644b8813f0a5f0"
I20260926 02:59:58.728636 5187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e9dcda72df1410c961756aee4c58387" candidate_uuid: "015a206faa844711a653aff5c570aa86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6299eb936744c0a995b71bbb4dcf4e"
I20260926 02:59:58.728720 5317 raft_consensus.cc:3037] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:58.728850 5187 raft_consensus.cc:3037] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:58.730975 5317 raft_consensus.cc:2454] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 015a206faa844711a653aff5c570aa86 in term 1.
I20260926 02:59:58.731133 5187 raft_consensus.cc:2454] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 015a206faa844711a653aff5c570aa86 in term 1.
I20260926 02:59:58.731365 5380 leader_election.cc:302] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 015a206faa844711a653aff5c570aa86, a2bfe738289746c9b3644b8813f0a5f0; no voters:
I20260926 02:59:58.731782 5516 raft_consensus.cc:2781] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:58.732211 5516 raft_consensus.cc:686] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 1 LEADER]: Becoming Leader. State: Replica: 015a206faa844711a653aff5c570aa86, State: Running, Role: LEADER
I20260926 02:59:58.732668 5516 consensus_queue.cc:237] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } }
I20260926 02:59:58.737291 5044 catalog_manager.cc:5495] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 015a206faa844711a653aff5c570aa86 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "015a206faa844711a653aff5c570aa86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "015a206faa844711a653aff5c570aa86" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46789 } health_report { overall_health: HEALTHY } } }
I20260926 02:59:58.762104 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.760241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.0.97.3:46789)
0926 02:59:58.760393 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.760397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.760461 (+ 64us) server_negotiation.cc:407] Connection header received
0926 02:59:58.760648 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.760651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.760723 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.760805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.761678 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.761682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.761685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.761697 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.761722 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.761732 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.761826 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.761890 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.761929 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 02:59:58.770174 5187 raft_consensus.cc:1261] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 1 FOLLOWER]: Refusing update from remote peer 015a206faa844711a653aff5c570aa86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:58.770232 5317 raft_consensus.cc:1261] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 1 FOLLOWER]: Refusing update from remote peer 015a206faa844711a653aff5c570aa86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 02:59:58.771359 5520 consensus_queue.cc:1035] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6299eb936744c0a995b71bbb4dcf4e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:58.771785 5516 consensus_queue.cc:1035] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2bfe738289746c9b3644b8813f0a5f0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 02:59:58.781375 5523 mvcc.cc:205] Tried to move back new op lower bound from 7333443988552626176 to 7333443988413145088. Current Snapshot: MvccSnapshot[applied={T|T < 7333443988552626176}]
I20260926 02:59:58.787147 5524 mvcc.cc:205] Tried to move back new op lower bound from 7333443988552626176 to 7333443988413145088. Current Snapshot: MvccSnapshot[applied={T|T < 7333443988552626176 or (T in {7333443988552626176})}]
I20260926 02:59:58.788074 5525 mvcc.cc:205] Tried to move back new op lower bound from 7333443988552626176 to 7333443988413145088. Current Snapshot: MvccSnapshot[applied={T|T < 7333443988552626176}]
I20260926 02:59:58.798925 5044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:58.799196 5044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 02:59:58.802696 5044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8cc9ef989b23476dba86fe6b4c852099: Sending DeleteTablet for 3 replicas of tablet 3e9dcda72df1410c961756aee4c58387
I20260926 02:59:58.803736 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9dcda72df1410c961756aee4c58387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:58 UTC) from {username='slave'} at 127.0.0.1:53660
I20260926 02:59:58.803845 5167 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9dcda72df1410c961756aee4c58387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:58 UTC) from {username='slave'} at 127.0.0.1:34058
I20260926 02:59:58.804208 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e9dcda72df1410c961756aee4c58387 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 02:59:58 UTC) from {username='slave'} at 127.0.0.1:40698
I20260926 02:59:58.805063 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5105
I20260926 02:59:58.816251 5533 tablet_replica.cc:330] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: stopping tablet replica
I20260926 02:59:58.816829 5533 raft_consensus.cc:2229] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:58.817495 5533 raft_consensus.cc:2258] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:58.818984 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.806423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.0.97.1:41505)
0926 02:59:58.806649 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.806659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.816398 (+ 9739us) server_negotiation.cc:407] Connection header received
0926 02:59:58.816810 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.816818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.816921 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.817036 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.818154 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.818159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.818163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.818186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.818223 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.818247 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.818358 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.818623 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.818685 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:59:58.819067 5535 tablet_replica.cc:330] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: stopping tablet replica
I20260926 02:59:58.819437 5535 raft_consensus.cc:2229] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 1 LEADER]: Raft consensus shutting down.
I20260926 02:59:58.820183 5535 raft_consensus.cc:2258] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:58.819624 5227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:58.819034 5534 tablet_replica.cc:330] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e: stopping tablet replica
I20260926 02:59:58.822167 5533 ts_tablet_manager.cc:1905] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:58.822443 5535 ts_tablet_manager.cc:1905] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 02:59:58.828963 5533 ts_tablet_manager.cc:1918] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:58.829145 5533 log.cc:1196] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/3e9dcda72df1410c961756aee4c58387
I20260926 02:59:58.829686 5533 ts_tablet_manager.cc:1939] T 3e9dcda72df1410c961756aee4c58387 P a2bfe738289746c9b3644b8813f0a5f0: Deleting consensus metadata
I20260926 02:59:58.831424 5030 catalog_manager.cc:4841] TS a2bfe738289746c9b3644b8813f0a5f0 (127.0.97.2:43651): tablet 3e9dcda72df1410c961756aee4c58387 (table test-table [id=87205565445448eead3965d8ddc660e1]) successfully deleted
I20260926 02:59:58.831769 5535 ts_tablet_manager.cc:1918] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 02:59:58.831940 5535 log.cc:1196] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/3e9dcda72df1410c961756aee4c58387
I20260926 02:59:58.832643 5535 ts_tablet_manager.cc:1939] T 3e9dcda72df1410c961756aee4c58387 P 015a206faa844711a653aff5c570aa86: Deleting consensus metadata
I20260926 02:59:58.834487 5032 catalog_manager.cc:4841] TS 015a206faa844711a653aff5c570aa86 (127.0.97.3:46789): tablet 3e9dcda72df1410c961756aee4c58387 (table test-table [id=87205565445448eead3965d8ddc660e1]) successfully deleted
I20260926 02:59:58.821420 5534 raft_consensus.cc:2229] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 02:59:58.924357 5534 raft_consensus.cc:2258] T 3e9dcda72df1410c961756aee4c58387 P 1f6299eb936744c0a995b71bbb4dcf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 02:59:58.924605 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5105
W20260926 02:59:58.947633 5030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41505 (error 108)
I20260926 02:59:58.948504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5235
I20260926 02:59:58.949966 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.949122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34078)
0926 02:59:58.949742 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:58.949833 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522}
W20260926 02:59:58.950235 5030 catalog_manager.cc:4584] TS 1f6299eb936744c0a995b71bbb4dcf4e (127.0.97.1:41505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e9dcda72df1410c961756aee4c58387: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
I20260926 02:59:58.951659 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:58.949561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.0.97.2:43651)
0926 02:59:58.949762 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:58.949769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:58.949788 (+ 19us) server_negotiation.cc:407] Connection header received
0926 02:59:58.949849 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:58.949855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:58.949952 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:58.950051 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:58.951060 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:58.951064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:58.951068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:58.951082 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:58.951111 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:58.951122 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:58.951221 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:58.951336 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 02:59:58.951378 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 02:59:58.952373 5357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:59.003793 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.003317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34090)
0926 02:59:59.003545 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:59.003660 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 02:59:59.004930 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.004586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34104)
0926 02:59:59.004732 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:59.004800 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 02:59:59.065565 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.065031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34114)
0926 02:59:59.065282 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:59.065390 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 02:59:59.066751 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.066334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34122)
0926 02:59:59.066495 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:59.066592 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 02:59:59.083221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5235
I20260926 02:59:59.102854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5365
I20260926 02:59:59.113212 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.107321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.0.97.3:46789)
0926 02:59:59.107544 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:59.107550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:59.107570 (+ 20us) server_negotiation.cc:407] Connection header received
0926 02:59:59.107692 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:59.107699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:59.107792 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:59.107915 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:59.112470 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:59.112474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:59.112477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:59.112492 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:59.112521 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:59.112532 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:59.112702 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:59.112803 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 02:59:59.112876 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 02:59:59.113978 5487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:59.147677 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.147137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34138)
0926 02:59:59.147438 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:59.147529 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260926 02:59:59.148664 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.148313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41505 (local address 127.0.0.1:34150)
0926 02:59:59.148500 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:59.148559 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 02:59:59.227955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5365
I20260926 02:59:59.242638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5015
I20260926 02:59:59.243758 5076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 02:59:59.345894 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5015
2026-09-26T02:59:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2601 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 02:59:59.386428 388 mini_oidc.cc:152] Starting JWKS server
I20260926 02:59:59.388917 388 webserver.cc:460] Webserver started at https://127.0.0.1:43891/ using document root <none> and password file <none>
I20260926 02:59:59.389261 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 02:59:59.389905 388 webserver.cc:460] Webserver started at http://127.0.0.1:35875/ using document root <none> and password file <none>
2026-09-26T02:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T02:59:59Z Disabled control of system clock
I20260926 02:59:59.415453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35255
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35255
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43891/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/testchainca.cert with env {}
W20260926 02:59:59.724222 5547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 02:59:59.724490 5547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 02:59:59.724573 5547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 02:59:59.731563 5547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 02:59:59.731658 5547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 02:59:59.731698 5547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 02:59:59.731781 5547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 02:59:59.731815 5547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 02:59:59.731865 5547 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 02:59:59.731904 5547 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43891/jwks.json
I20260926 02:59:59.741431 5547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43891/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 02:59:59.742594 5547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 02:59:59.743799 5547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 02:59:59.755327 5555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:59.755366 5553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 02:59:59.755403 5552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 02:59:59.756059 5547 server_base.cc:1034] running on GCE node
I20260926 02:59:59.756783 5547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 02:59:59.758174 5547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 02:59:59.759380 5547 hybrid_clock.cc:648] HybridClock initialized: now 1790391599759341 us; error 28 us; skew 500 ppm
I20260926 02:59:59.759737 5547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 02:59:59.762090 5547 webserver.cc:460] Webserver started at http://127.0.97.62:40967/ using document root <none> and password file <none>
I20260926 02:59:59.762748 5547 fs_manager.cc:359] Metadata directory not provided
I20260926 02:59:59.762866 5547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 02:59:59.763177 5547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 02:59:59.766048 5547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "5800c1f7b6974c80ae405b5b1f5f2949"
format_stamp: "Formatted at 2026-09-26 02:59:59 on dist-test-slave-fccw"
server_key: "66b95c55ff1cdda366f6ab6049635f8b"
server_key_iv: "253d4c505057d909f02e0afb55b2faa7"
server_key_version: "encryptionkey@0"
I20260926 02:59:59.766810 5547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "5800c1f7b6974c80ae405b5b1f5f2949"
format_stamp: "Formatted at 2026-09-26 02:59:59 on dist-test-slave-fccw"
server_key: "66b95c55ff1cdda366f6ab6049635f8b"
server_key_iv: "253d4c505057d909f02e0afb55b2faa7"
server_key_version: "encryptionkey@0"
I20260926 02:59:59.771705 5547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 02:59:59.774986 5561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 02:59:59.776225 5547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 02:59:59.776398 5547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "5800c1f7b6974c80ae405b5b1f5f2949"
format_stamp: "Formatted at 2026-09-26 02:59:59 on dist-test-slave-fccw"
server_key: "66b95c55ff1cdda366f6ab6049635f8b"
server_key_iv: "253d4c505057d909f02e0afb55b2faa7"
server_key_version: "encryptionkey@0"
I20260926 02:59:59.776610 5547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 02:59:59.792146 5547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 02:59:59.792913 5547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 02:59:59.793210 5547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 02:59:59.810544 5547 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35255
I20260926 02:59:59.810565 5612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35255 every 8 connection(s)
I20260926 02:59:59.811868 5547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 02:59:59.815258 5613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 02:59:59.821949 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5547
I20260926 02:59:59.822182 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 02:59:59.822763 388 external_mini_cluster.cc:1428] Setting key 4c93767fd536f7894cdc814a634975a1
I20260926 02:59:59.824959 5613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949: Bootstrap starting.
I20260926 02:59:59.828171 5613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949: Neither blocks nor log segments found. Creating new log.
I20260926 02:59:59.829396 5613 log.cc:824] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949: Log is configured to *not* fsync() on all Append() calls
I20260926 02:59:59.832435 5613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949: No bootstrap required, opened a new log
I20260926 02:59:59.836489 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.824571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.0.97.62:35255)
0926 02:59:59.825594 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:59.825605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:59.825647 (+ 42us) server_negotiation.cc:407] Connection header received
0926 02:59:59.827337 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:59.827359 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:59.827788 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:59.828097 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 02:59:59.828746 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:59.830379 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:59.832248 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:59.832950 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:59.833179 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:59.834307 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 02:59:59.834314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 02:59:59.834327 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 02:59:59.834862 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0926 02:59:59.834940 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 02:59:59.834959 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 02:59:59.835038 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 02:59:59.835282 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 02:59:59.835606 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":376,"threads_started":1}
I20260926 02:59:59.840145 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 02:59:59.840693 5613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35255 } }
I20260926 02:59:59.841027 5613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 02:59:59.841156 5613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5800c1f7b6974c80ae405b5b1f5f2949, State: Initialized, Role: FOLLOWER
I20260926 02:59:59.841735 5613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35255 } }
I20260926 02:59:59.841949 5613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 02:59:59.842065 5613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 02:59:59.842289 5613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 0 FOLLOWER]: Advancing to term 1
I20260926 02:59:59.844636 5613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35255 } }
I20260926 02:59:59.845105 5613 leader_election.cc:302] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5800c1f7b6974c80ae405b5b1f5f2949; no voters:
I20260926 02:59:59.845628 5613 leader_election.cc:288] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 02:59:59.845783 5618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 02:59:59.846410 5618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [term 1 LEADER]: Becoming Leader. State: Replica: 5800c1f7b6974c80ae405b5b1f5f2949, State: Running, Role: LEADER
I20260926 02:59:59.846927 5618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35255 } }
I20260926 02:59:59.847549 5613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 02:59:59.849838 5619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35255 } } }
I20260926 02:59:59.849862 5620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5800c1f7b6974c80ae405b5b1f5f2949. Latest consensus state: current_term: 1 leader_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800c1f7b6974c80ae405b5b1f5f2949" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35255 } } }
I20260926 02:59:59.850239 5619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:59.850277 5620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949 [sys.catalog]: This master's current role is: LEADER
I20260926 02:59:59.851897 5623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 02:59:59.856448 5623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 02:59:59.864545 5623 catalog_manager.cc:1347] Generated new cluster ID: 43371227f9f2485caf292f493b202fb5
I20260926 02:59:59.864641 5623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 02:59:59.891651 5623 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 02:59:59.892834 5623 catalog_manager.cc:1504] Loading token signing keys...
I20260926 02:59:59.901613 5623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5800c1f7b6974c80ae405b5b1f5f2949: Generated new TSK 0
I20260926 02:59:59.902563 5623 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 02:59:59.958858 5643 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 02:59:59.959012 5643 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 02:59:59.960175 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 02:59:59.931578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.0.97.62:35255)
0926 02:59:59.931831 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 02:59:59.931838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 02:59:59.932291 (+ 453us) server_negotiation.cc:407] Connection header received
0926 02:59:59.932591 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 02:59:59.932595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 02:59:59.932665 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 02:59:59.932787 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0926 02:59:59.933453 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:59.934203 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:59.936229 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 02:59:59.936635 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 02:59:59.936722 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:59.937328 (+ 606us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 02:59:59.959406 (+ 22078us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:59:59.959838 (+ 432us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 02:59:59.960294 5616 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 02:59:59.962538 5642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 02:59:59.931491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35255 (local address 127.0.0.1:37442)
0926 02:59:59.932171 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 02:59:59.932196 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 02:59:59.932383 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 02:59:59.932843 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 02:59:59.932850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 02:59:59.932863 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 02:59:59.933293 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:59.933305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:59.934324 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:59.934330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:59.936095 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 02:59:59.936112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 02:59:59.936743 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 02:59:59.936807 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 02:59:59.936987 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 02:59:59.937098 (+ 111us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 02:59:59.959890 (+ 22792us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 02:59:59.960790 (+ 900us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":326,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:35255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 02:59:59.964457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5547
I20260926 02:59:59.965759 5608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:00.059034 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5547
2026-09-26T03:00:00Z chronyd exiting
I20260926 03:00:00.108887 388 test_util.cc:175] -----------------------------------------------
I20260926 03:00:00.109057 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790391557890803-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (726 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 03:00:00.112447 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:00:00.114256 388 webserver.cc:460] Webserver started at https://127.0.0.1:40435/ using document root <none> and password file <none>
I20260926 03:00:00.114554 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:00:00.115199 388 webserver.cc:460] Webserver started at http://127.0.0.1:36939/ using document root <none> and password file <none>
2026-09-26T03:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:00Z Disabled control of system clock
I20260926 03:00:00.139549 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34907
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34907
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40435/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/testchainca.cert with env {}
W20260926 03:00:00.444088 5650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:00.444361 5650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:00.444443 5650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:00.451462 5650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:00.451560 5650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:00.451615 5650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:00.451665 5650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:00.451702 5650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:00:00.451764 5650 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:00:00.451803 5650 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40435/jwks.json
I20260926 03:00:00.461316 5650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40435/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:00.462496 5650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:00.463701 5650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:00.474957 5658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:00.474998 5655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:00.475000 5656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:00.475854 5650 server_base.cc:1034] running on GCE node
I20260926 03:00:00.476545 5650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:00.477917 5650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:00.479111 5650 hybrid_clock.cc:648] HybridClock initialized: now 1790391600479075 us; error 31 us; skew 500 ppm
I20260926 03:00:00.479435 5650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:00.481516 5650 webserver.cc:460] Webserver started at http://127.0.97.62:37533/ using document root <none> and password file <none>
I20260926 03:00:00.482163 5650 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:00.482273 5650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:00.482579 5650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:00.485340 5650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "17ba702e24ed4f138d7ceac753792a51"
format_stamp: "Formatted at 2026-09-26 03:00:00 on dist-test-slave-fccw"
server_key: "6cc006d2861ef4592f304d7c95e2a62d"
server_key_iv: "82b49af897389c27ed59f8e6db51a50c"
server_key_version: "encryptionkey@0"
I20260926 03:00:00.486088 5650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "17ba702e24ed4f138d7ceac753792a51"
format_stamp: "Formatted at 2026-09-26 03:00:00 on dist-test-slave-fccw"
server_key: "6cc006d2861ef4592f304d7c95e2a62d"
server_key_iv: "82b49af897389c27ed59f8e6db51a50c"
server_key_version: "encryptionkey@0"
I20260926 03:00:00.490680 5650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:00:00.493808 5664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:00.495041 5650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:00.495219 5650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "17ba702e24ed4f138d7ceac753792a51"
format_stamp: "Formatted at 2026-09-26 03:00:00 on dist-test-slave-fccw"
server_key: "6cc006d2861ef4592f304d7c95e2a62d"
server_key_iv: "82b49af897389c27ed59f8e6db51a50c"
server_key_version: "encryptionkey@0"
I20260926 03:00:00.495414 5650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:00.519147 5650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:00.519943 5650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:00.520219 5650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:00.537899 5650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34907
I20260926 03:00:00.537931 5715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34907 every 8 connection(s)
I20260926 03:00:00.539143 5650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:00.542608 5716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:00.545454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5650
I20260926 03:00:00.545643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:00.546250 388 external_mini_cluster.cc:1428] Setting key 46ea2cf8ac34de73051a6756bfc88c07
I20260926 03:00:00.555382 5716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51: Bootstrap starting.
I20260926 03:00:00.559260 5716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:00.560535 5716 log.cc:824] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:00.560774 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:00.548072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.0.97.62:34907)
0926 03:00:00.549098 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:00.549112 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:00.549161 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:00:00.551026 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:00.551057 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:00.551650 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:00.552101 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:00.552632 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:00.554536 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:00.555807 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:00.556792 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:00.557047 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:00.558085 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:00.558097 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:00.558117 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:00.558794 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:00.558903 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:00.558928 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:00.559054 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:00.559217 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:00:00.559532 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":306,"threads_started":1}
I20260926 03:00:00.563403 5716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51: No bootstrap required, opened a new log
I20260926 03:00:00.564437 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:00:00.570784 5716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ba702e24ed4f138d7ceac753792a51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34907 } }
I20260926 03:00:00.571095 5716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:00.571187 5716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ba702e24ed4f138d7ceac753792a51, State: Initialized, Role: FOLLOWER
I20260926 03:00:00.571766 5716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ba702e24ed4f138d7ceac753792a51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34907 } }
I20260926 03:00:00.571988 5716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:00.572137 5716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:00.572300 5716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:00.574713 5716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ba702e24ed4f138d7ceac753792a51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34907 } }
I20260926 03:00:00.575191 5716 leader_election.cc:302] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17ba702e24ed4f138d7ceac753792a51; no voters:
I20260926 03:00:00.575709 5716 leader_election.cc:288] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:00.575851 5721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:00.576551 5721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [term 1 LEADER]: Becoming Leader. State: Replica: 17ba702e24ed4f138d7ceac753792a51, State: Running, Role: LEADER
I20260926 03:00:00.577108 5721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ba702e24ed4f138d7ceac753792a51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34907 } }
I20260926 03:00:00.577842 5716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:00.580677 5722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17ba702e24ed4f138d7ceac753792a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ba702e24ed4f138d7ceac753792a51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34907 } } }
I20260926 03:00:00.580672 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17ba702e24ed4f138d7ceac753792a51. Latest consensus state: current_term: 1 leader_uuid: "17ba702e24ed4f138d7ceac753792a51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ba702e24ed4f138d7ceac753792a51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34907 } } }
I20260926 03:00:00.581352 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:00.581373 5722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:00.588414 5731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:00.592330 5731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:00.599786 5731 catalog_manager.cc:1347] Generated new cluster ID: f5610587dbc74f519483010798f24c9a
I20260926 03:00:00.599907 5731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:00.616602 5731 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:00.617834 5731 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:00.626652 5731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17ba702e24ed4f138d7ceac753792a51: Generated new TSK 0
I20260926 03:00:00.627357 5731 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:00:00.662436 5746 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 03:00:00.662549 5746 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:00:00.663610 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:00.644373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.0.97.62:34907)
0926 03:00:00.644572 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:00.644578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:00.644980 (+ 402us) server_negotiation.cc:407] Connection header received
0926 03:00:00.645245 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:00.645250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:00.645332 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:00.645482 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:00:00.645900 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:00.647007 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:00.648294 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:00.648956 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:00.649125 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:00.649503 (+ 378us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:00:00.663016 (+ 13513us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:00:00.663349 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 03:00:00.663625 5745 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:00.644247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34907 (local address 127.0.0.1:34762)
0926 03:00:00.644886 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:00.644901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:00.645062 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:00.645504 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:00.645507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:00.645516 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:00:00.645765 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:00.645773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:00.647129 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:00.647133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:00.648158 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:00.648168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:00.649104 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:00.649137 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:00.649254 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:00.649282 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:00:00.663200 (+ 13918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:00:00.663364 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34907: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":369,"threads_started":1}
W20260926 03:00:00.663725 5719 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40435/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 03:00:00.665156 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5650
I20260926 03:00:00.666131 5711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:00.753769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5650
2026-09-26T03:00:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (695 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 03:00:00.808336 388 mini_oidc.cc:152] Starting JWKS server
I20260926 03:00:00.810110 388 webserver.cc:460] Webserver started at https://127.0.0.1:43669/ using document root <none> and password file <none>
I20260926 03:00:00.810297 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:00:00.810798 388 webserver.cc:460] Webserver started at http://127.0.0.1:40317/ using document root <none> and password file <none>
2026-09-26T03:00:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:00Z Disabled control of system clock
I20260926 03:00:00.834635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38919
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38919
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43669/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:00:01.143693 5753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:01.143992 5753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:01.144093 5753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:01.151085 5753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:01.151203 5753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:01.151299 5753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:01.151396 5753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:01.151473 5753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:00:01.151561 5753 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:00:01.151615 5753 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43669/jwks.json
I20260926 03:00:01.160460 5753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43669/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:01.161705 5753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:01.163458 5753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:01.174482 5761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:01.174602 5758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:01.174675 5759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:01.175134 5753 server_base.cc:1034] running on GCE node
I20260926 03:00:01.175891 5753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:01.177285 5753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:01.178504 5753 hybrid_clock.cc:648] HybridClock initialized: now 1790391601178468 us; error 47 us; skew 500 ppm
I20260926 03:00:01.178834 5753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:01.180898 5753 webserver.cc:460] Webserver started at http://127.0.97.62:45109/ using document root <none> and password file <none>
I20260926 03:00:01.181600 5753 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:01.181722 5753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:01.182035 5753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:01.184710 5753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "1f940b7e1e094999a9b783994ee202b0"
format_stamp: "Formatted at 2026-09-26 03:00:01 on dist-test-slave-fccw"
server_key: "0d1da841bb7375a08eadfa06066a6acc"
server_key_iv: "2d40869c34f792e6f475c4d147ad947d"
server_key_version: "encryptionkey@0"
I20260926 03:00:01.185472 5753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "1f940b7e1e094999a9b783994ee202b0"
format_stamp: "Formatted at 2026-09-26 03:00:01 on dist-test-slave-fccw"
server_key: "0d1da841bb7375a08eadfa06066a6acc"
server_key_iv: "2d40869c34f792e6f475c4d147ad947d"
server_key_version: "encryptionkey@0"
I20260926 03:00:01.190150 5753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:00:01.193245 5767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:01.194466 5753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:00:01.194657 5753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "1f940b7e1e094999a9b783994ee202b0"
format_stamp: "Formatted at 2026-09-26 03:00:01 on dist-test-slave-fccw"
server_key: "0d1da841bb7375a08eadfa06066a6acc"
server_key_iv: "2d40869c34f792e6f475c4d147ad947d"
server_key_version: "encryptionkey@0"
I20260926 03:00:01.194855 5753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:01.212194 5753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:01.212914 5753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:01.213205 5753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:01.230437 5753 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38919
I20260926 03:00:01.230455 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38919 every 8 connection(s)
I20260926 03:00:01.231673 5753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:01.235087 5819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:01.240007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5753
I20260926 03:00:01.240224 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:01.240813 388 external_mini_cluster.cc:1428] Setting key 2737826b91595f8aa487d02c2c4040e6
I20260926 03:00:01.244931 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0: Bootstrap starting.
I20260926 03:00:01.248121 5819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:01.249305 5819 log.cc:824] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:01.252466 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0: No bootstrap required, opened a new log
I20260926 03:00:01.254107 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:01.242673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.0.97.62:38919)
0926 03:00:01.243594 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:01.243603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:01.243641 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:00:01.245213 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:01.245233 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:01.245634 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:01.245924 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:01.246524 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.247733 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.249627 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.250738 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.250889 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:01.251968 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:01.251974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:01.251986 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:01.252461 (+ 475us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:01.252544 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:01.252593 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:01.252697 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:01.252902 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 03:00:01.253247 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":368,"threads_started":1}
I20260926 03:00:01.257732 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:00:01.261210 5819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f940b7e1e094999a9b783994ee202b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38919 } }
I20260926 03:00:01.261567 5819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:01.261672 5819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f940b7e1e094999a9b783994ee202b0, State: Initialized, Role: FOLLOWER
I20260926 03:00:01.262243 5819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f940b7e1e094999a9b783994ee202b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38919 } }
I20260926 03:00:01.262492 5819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:01.262633 5819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:01.262807 5819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:01.264801 5819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f940b7e1e094999a9b783994ee202b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38919 } }
I20260926 03:00:01.265296 5819 leader_election.cc:302] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f940b7e1e094999a9b783994ee202b0; no voters:
I20260926 03:00:01.265856 5819 leader_election.cc:288] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:01.265973 5824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:01.266629 5824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [term 1 LEADER]: Becoming Leader. State: Replica: 1f940b7e1e094999a9b783994ee202b0, State: Running, Role: LEADER
I20260926 03:00:01.267132 5824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f940b7e1e094999a9b783994ee202b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38919 } }
I20260926 03:00:01.267761 5819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:01.269786 5826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f940b7e1e094999a9b783994ee202b0. Latest consensus state: current_term: 1 leader_uuid: "1f940b7e1e094999a9b783994ee202b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f940b7e1e094999a9b783994ee202b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38919 } } }
I20260926 03:00:01.270205 5826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:01.270766 5825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f940b7e1e094999a9b783994ee202b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f940b7e1e094999a9b783994ee202b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38919 } } }
I20260926 03:00:01.271127 5825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:01.271912 5832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:01.277019 5832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:01.284523 5832 catalog_manager.cc:1347] Generated new cluster ID: eab51d74a2d24fd9a93dd9d852c7dee1
I20260926 03:00:01.284634 5832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:01.302625 5832 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:01.303723 5832 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:01.310397 5832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f940b7e1e094999a9b783994ee202b0: Generated new TSK 0
I20260926 03:00:01.311285 5832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:00:01.332243 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:01.324205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.0.97.62:38919)
0926 03:00:01.324413 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:01.324421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:01.324799 (+ 378us) server_negotiation.cc:407] Connection header received
0926 03:00:01.325034 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:01.325040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:01.325126 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:01.325236 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:00:01.325763 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.326449 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.328466 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.328860 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.328950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:01.329507 (+ 557us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:00:01.331638 (+ 2131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:00:01.331938 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 03:00:01.332347 5822 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 03:00:01.334414 5848 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:01.324084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38919 (local address 127.0.0.1:58990)
0926 03:00:01.324694 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:01.324716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:01.324859 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:01.325280 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:01.325288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:01.325302 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:00:01.325624 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:01.325636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.326592 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.326598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:01.328262 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:01.328276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.328987 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.329075 (+ 88us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:01.329264 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:01.329305 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:00:01.331837 (+ 2532us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:00:01.333971 (+ 2134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:38919: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43669/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":1843,"thread_start_us":318,"threads_started":1}
I20260926 03:00:01.335505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5753
I20260926 03:00:01.336526 5814 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 03:00:01.344555 5849 webserver.cc:533] Webserver: sslize failed:
E20260926 03:00:01.344684 5849 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:00:01.425113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5753
2026-09-26T03:00:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (670 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T03:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:01Z Disabled control of system clock
I20260926 03:00:01.503599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42635
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42635
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/testchainca.cert with env {}
W20260926 03:00:01.816082 5854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:01.816372 5854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:01.816550 5854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:01.823647 5854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:01.823740 5854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:01.823812 5854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:01.823911 5854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:01.823987 5854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:00:01.824101 5854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:00:01.824211 5854 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 03:00:01.834059 5854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:01.835333 5854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:01.836613 5854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:01.848292 5859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:01.848330 5862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:01.848248 5860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:01.849265 5854 server_base.cc:1034] running on GCE node
I20260926 03:00:01.849977 5854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:01.851277 5854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:01.852502 5854 hybrid_clock.cc:648] HybridClock initialized: now 1790391601852451 us; error 45 us; skew 500 ppm
I20260926 03:00:01.852852 5854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:01.855070 5854 webserver.cc:460] Webserver started at http://127.0.97.62:37369/ using document root <none> and password file <none>
I20260926 03:00:01.855751 5854 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:01.855867 5854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:01.856187 5854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:01.859050 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "f3152231029b40c98c6de3e6d4d1597e"
format_stamp: "Formatted at 2026-09-26 03:00:01 on dist-test-slave-fccw"
server_key: "e82382c45fc3201893bb5e4f92698603"
server_key_iv: "1f5f8536b44b22548ac6f734c22d2842"
server_key_version: "encryptionkey@0"
I20260926 03:00:01.859834 5854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "f3152231029b40c98c6de3e6d4d1597e"
format_stamp: "Formatted at 2026-09-26 03:00:01 on dist-test-slave-fccw"
server_key: "e82382c45fc3201893bb5e4f92698603"
server_key_iv: "1f5f8536b44b22548ac6f734c22d2842"
server_key_version: "encryptionkey@0"
I20260926 03:00:01.864677 5854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260926 03:00:01.867986 5868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:01.869304 5854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:01.869467 5854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "f3152231029b40c98c6de3e6d4d1597e"
format_stamp: "Formatted at 2026-09-26 03:00:01 on dist-test-slave-fccw"
server_key: "e82382c45fc3201893bb5e4f92698603"
server_key_iv: "1f5f8536b44b22548ac6f734c22d2842"
server_key_version: "encryptionkey@0"
I20260926 03:00:01.869666 5854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:01.893329 5854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:01.894109 5854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:01.894385 5854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:01.912108 5854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42635
I20260926 03:00:01.912114 5919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42635 every 8 connection(s)
I20260926 03:00:01.913479 5854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:01.916960 5920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:01.920285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5854
I20260926 03:00:01.920469 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:01.921303 388 external_mini_cluster.cc:1428] Setting key c209a8ee75e90a32b9917465b843ac29
I20260926 03:00:01.928678 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e: Bootstrap starting.
I20260926 03:00:01.931674 5920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:01.932696 5920 log.cc:824] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:01.933728 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:01.923155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.0.97.62:42635)
0926 03:00:01.924017 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:01.924027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:01.924077 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:00:01.925655 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:01.925676 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:01.926135 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:01.926509 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:01.927081 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.928221 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.929518 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:01.930184 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:01.930398 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:01.931487 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:01.931493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:01.931506 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:01.932049 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:01.932133 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:01.932151 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:01.932269 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:01.932484 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 03:00:01.932744 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":286,"threads_started":1}
I20260926 03:00:01.935295 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e: No bootstrap required, opened a new log
I20260926 03:00:01.937137 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:00:01.942790 5920 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3152231029b40c98c6de3e6d4d1597e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42635 } }
I20260926 03:00:01.943109 5920 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:01.943220 5920 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3152231029b40c98c6de3e6d4d1597e, State: Initialized, Role: FOLLOWER
I20260926 03:00:01.943862 5920 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3152231029b40c98c6de3e6d4d1597e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42635 } }
I20260926 03:00:01.944106 5920 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:01.944267 5920 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:01.944438 5920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:01.946491 5920 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3152231029b40c98c6de3e6d4d1597e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42635 } }
I20260926 03:00:01.946939 5920 leader_election.cc:302] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3152231029b40c98c6de3e6d4d1597e; no voters:
I20260926 03:00:01.947467 5920 leader_election.cc:288] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:01.947679 5925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:01.948750 5925 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [term 1 LEADER]: Becoming Leader. State: Replica: f3152231029b40c98c6de3e6d4d1597e, State: Running, Role: LEADER
I20260926 03:00:01.949260 5925 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3152231029b40c98c6de3e6d4d1597e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42635 } }
I20260926 03:00:01.949631 5920 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:01.951803 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3152231029b40c98c6de3e6d4d1597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3152231029b40c98c6de3e6d4d1597e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42635 } } }
I20260926 03:00:01.952147 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:01.951982 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3152231029b40c98c6de3e6d4d1597e. Latest consensus state: current_term: 1 leader_uuid: "f3152231029b40c98c6de3e6d4d1597e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3152231029b40c98c6de3e6d4d1597e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42635 } } }
I20260926 03:00:01.952350 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:01.953725 5934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:01.958673 5934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:01.968190 5934 catalog_manager.cc:1347] Generated new cluster ID: d3d577d3507d4eb7be4d74dcf3b4128b
I20260926 03:00:01.968325 5934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:01.987711 5934 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:01.988921 5934 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:01.996385 5934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3152231029b40c98c6de3e6d4d1597e: Generated new TSK 0
I20260926 03:00:01.997221 5934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:00:02.062358 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:02.008837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.0.97.62:42635)
0926 03:00:02.009019 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:02.009026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:02.009571 (+ 545us) server_negotiation.cc:407] Connection header received
0926 03:00:02.009783 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:02.009786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:02.009850 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:02.009966 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:00:02.010481 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:02.011146 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:02.013144 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:02.013554 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:02.013643 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:02.014188 (+ 545us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:00:02.061810 (+ 47622us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:00:02.062132 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54790: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 03:00:02.062556 5923 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54790: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 03:00:02.063167 5949 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:02.008661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42635 (local address 127.0.0.1:54790)
0926 03:00:02.009460 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:02.009481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:02.009628 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:02.010012 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:02.010019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:02.010031 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:00:02.010346 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:02.010357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:02.011285 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:02.011292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:02.012996 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:02.013037 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:02.013680 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:02.013730 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:02.013910 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:02.013953 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:00:02.062094 (+ 48141us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:00:02.062430 (+ 336us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42635: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":506,"threads_started":1}
I20260926 03:00:02.064527 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5854
I20260926 03:00:02.065661 5915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:02.159768 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5854
2026-09-26T03:00:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (714 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (394 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (392 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:00:03 dist-test-slave-fccw krb5kdc[5964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:00:03 dist-test-slave-fccw krb5kdc[5964](info): set up 2 sockets
Sep 26 03:00:03 dist-test-slave-fccw krb5kdc[5964](info): commencing operation
krb5kdc: starting...
W20260926 03:00:04.037798 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:00:04 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:00:04.231802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34977
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:04.541764 5977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:04.542096 5977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:04.542253 5977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:04.549269 5977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:04.549400 5977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:04.549502 5977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:04.549604 5977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:04.549695 5977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:04.559290 5977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:04.560595 5977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:04.561892 5977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:04.573621 5982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:04.573693 5983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:04.573669 5985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:04.574720 5977 server_base.cc:1034] running on GCE node
I20260926 03:00:04.575466 5977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:04.576766 5977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:04.578007 5977 hybrid_clock.cc:648] HybridClock initialized: now 1790391604577966 us; error 32 us; skew 500 ppm
I20260926 03:00:04.578305 5977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:04 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:04.588050 5977 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:00:04.589635 5977 webserver.cc:460] Webserver started at http://127.0.97.62:44935/ using document root <none> and password file <none>
I20260926 03:00:04.590299 5977 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:04.590399 5977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:04.590660 5977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:04.593564 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "069ab5b0e2684b1292faf24a04123587"
format_stamp: "Formatted at 2026-09-26 03:00:04 on dist-test-slave-fccw"
server_key: "b52b9ab8a0add0bc59ee91ac13c69838"
server_key_iv: "34917689fc1deeedcb2988f375f19553"
server_key_version: "encryptionkey@0"
I20260926 03:00:04.594265 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "069ab5b0e2684b1292faf24a04123587"
format_stamp: "Formatted at 2026-09-26 03:00:04 on dist-test-slave-fccw"
server_key: "b52b9ab8a0add0bc59ee91ac13c69838"
server_key_iv: "34917689fc1deeedcb2988f375f19553"
server_key_version: "encryptionkey@0"
I20260926 03:00:04.600507 5977 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:00:04.606019 5992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:04.607887 5977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:00:04.608251 5977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "069ab5b0e2684b1292faf24a04123587"
format_stamp: "Formatted at 2026-09-26 03:00:04 on dist-test-slave-fccw"
server_key: "b52b9ab8a0add0bc59ee91ac13c69838"
server_key_iv: "34917689fc1deeedcb2988f375f19553"
server_key_version: "encryptionkey@0"
I20260926 03:00:04.608471 5977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:04.644744 5977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:04.648798 5977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:04.649094 5977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:04.666141 5977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34977
I20260926 03:00:04.666177 6043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34977 every 8 connection(s)
I20260926 03:00:04.667435 5977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:04.669055 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 5977
I20260926 03:00:04.669252 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:04.669872 388 external_mini_cluster.cc:1428] Setting key 9f01b0928a87fa9673c4bb8639ecb212
I20260926 03:00:04.671559 6044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:00:04 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:04.684437 6044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587: Bootstrap starting.
I20260926 03:00:04.688475 6044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:04.689755 6044 log.cc:824] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:04.692451 6044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587: No bootstrap required, opened a new log
I20260926 03:00:04.692812 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:04.671769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.0.97.62:34977)
0926 03:00:04.672516 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:04.672531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:04.672581 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:00:04.674611 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:04.674636 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:04.675131 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:04.675539 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:04.676774 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:04.678815 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:04.679988 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:04.681055 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:04.681240 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:04.684922 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:04.684932 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:04.684949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:04.685016 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:04.688724 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:04.689458 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:04.689469 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:04.689480 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:04.689677 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:04.690063 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:04.690068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:04.690078 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:04.690706 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:04.691038 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:04.691453 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 03:00:04.691845 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":295,"threads_started":1}
I20260926 03:00:04.700789 6044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069ab5b0e2684b1292faf24a04123587" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260926 03:00:04.701186 6044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:04.701308 6044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069ab5b0e2684b1292faf24a04123587, State: Initialized, Role: FOLLOWER
I20260926 03:00:04.701947 6044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069ab5b0e2684b1292faf24a04123587" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260926 03:00:04.702185 6044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:04.702322 6044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:04.702492 6044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:04.704502 6044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069ab5b0e2684b1292faf24a04123587" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260926 03:00:04.705104 6044 leader_election.cc:302] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 069ab5b0e2684b1292faf24a04123587; no voters:
I20260926 03:00:04.705646 6044 leader_election.cc:288] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:04.705776 6049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:04.706617 6049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [term 1 LEADER]: Becoming Leader. State: Replica: 069ab5b0e2684b1292faf24a04123587, State: Running, Role: LEADER
I20260926 03:00:04.707214 6049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069ab5b0e2684b1292faf24a04123587" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } }
I20260926 03:00:04.707721 6044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:04.710028 6050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "069ab5b0e2684b1292faf24a04123587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069ab5b0e2684b1292faf24a04123587" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } } }
I20260926 03:00:04.710448 6050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:04.711066 6051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 069ab5b0e2684b1292faf24a04123587. Latest consensus state: current_term: 1 leader_uuid: "069ab5b0e2684b1292faf24a04123587" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "069ab5b0e2684b1292faf24a04123587" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34977 } } }
I20260926 03:00:04.711416 6051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:04.712167 6058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:04.717115 6058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:04.726580 6058 catalog_manager.cc:1347] Generated new cluster ID: 143f9d58a3f8493ea465fda81af837e3
I20260926 03:00:04.726698 6058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:04.743716 6058 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:04.744990 6058 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:04.760775 6058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 069ab5b0e2684b1292faf24a04123587: Generated new TSK 0
I20260926 03:00:04.761639 6058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:00:04.808188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:05.131235 6070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:05.131484 6070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:05.131659 6070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:05.138479 6070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:05.138594 6070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:05.138777 6070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:05.147648 6070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:05.148780 6070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:05.150033 6070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:05.162695 6078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:05.162657 6076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:05.162738 6075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:05.163467 6070 server_base.cc:1034] running on GCE node
I20260926 03:00:05.164009 6070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:05.164830 6070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:05.166029 6070 hybrid_clock.cc:648] HybridClock initialized: now 1790391605165994 us; error 69 us; skew 500 ppm
I20260926 03:00:05.166352 6070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:05 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:05.169370 6070 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:00:05.170768 6070 webserver.cc:460] Webserver started at http://127.0.97.1:37141/ using document root <none> and password file <none>
I20260926 03:00:05.171423 6070 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:05.171552 6070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:05.171865 6070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:05.174721 6070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "e1c483b826ef4e7ab9681eaf69f73c93"
format_stamp: "Formatted at 2026-09-26 03:00:05 on dist-test-slave-fccw"
server_key: "ed419105d33538511fcc228dc7ada861"
server_key_iv: "3443103144612e38281712c6b93287a2"
server_key_version: "encryptionkey@0"
I20260926 03:00:05.175491 6070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e1c483b826ef4e7ab9681eaf69f73c93"
format_stamp: "Formatted at 2026-09-26 03:00:05 on dist-test-slave-fccw"
server_key: "ed419105d33538511fcc228dc7ada861"
server_key_iv: "3443103144612e38281712c6b93287a2"
server_key_version: "encryptionkey@0"
I20260926 03:00:05.180348 6070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 03:00:05.183544 6085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:05.185214 6070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:05.185380 6070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "e1c483b826ef4e7ab9681eaf69f73c93"
format_stamp: "Formatted at 2026-09-26 03:00:05 on dist-test-slave-fccw"
server_key: "ed419105d33538511fcc228dc7ada861"
server_key_iv: "3443103144612e38281712c6b93287a2"
server_key_version: "encryptionkey@0"
I20260926 03:00:05.185586 6070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:05.204108 6070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:05.207963 6070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:05.208256 6070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:05.209265 6070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:05.210503 6070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:05.210614 6070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:05.210764 6070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:05.210829 6070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:05.243165 6070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35947
I20260926 03:00:05.243204 6197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35947 every 8 connection(s)
I20260926 03:00:05.244408 6070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:05.245975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6070
I20260926 03:00:05.246167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:00:05.246847 388 external_mini_cluster.cc:1428] Setting key c76bbb2ff91f127b35e608a7ed87824b
Sep 26 03:00:05 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:05.266690 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:05.246955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34017 (local address 127.0.97.62:34977)
0926 03:00:05.247438 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:05.247446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:05.248553 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 03:00:05.251268 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:05.251273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:05.251330 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:05.251418 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:05.253461 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.254130 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.256208 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.256794 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.256888 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:05.261296 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:05.261301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:05.261304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:05.261332 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:05.263774 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:05.264440 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:05.264445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:05.264455 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:05.264563 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:05.265016 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:05.265021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:05.265025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:05.265280 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:05.265428 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:05.266080 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 03:00:05.266338 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":231,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:00:05.267658 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:05.247522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.1:34017)
0926 03:00:05.248375 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:05.248425 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:05.250963 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:05.251691 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:05.251706 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:05.252465 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:05.253302 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:05.253328 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.254275 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.254283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:05.256067 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:05.256078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.257030 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.257040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:05.257376 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:05.258181 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:05.258213 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:05.261084 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:05.263907 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:05.263915 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:05.263931 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:05.264317 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:05.264697 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:05.264701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:05.264705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:05.264880 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:05.265403 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:05.265410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:05.265727 (+ 317us) client_negotiation.cc:769] Sending connection context
0926 03:00:05.265997 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 03:00:05.266514 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":338,"threads_started":1}
I20260926 03:00:05.269210 6198 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34977
I20260926 03:00:05.269555 6198 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:00:05.270364 6198 heartbeater.cc:511] Master 127.0.97.62:34977 requested a full tablet report, sending...
I20260926 03:00:05.272615 6009 ts_manager.cc:194] Registered new tserver with Master: e1c483b826ef4e7ab9681eaf69f73c93 (127.0.97.1:35947)
I20260926 03:00:05.274762 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:00:05.295796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:05.606482 6204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:05.606791 6204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:05.606886 6204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:05.614045 6204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:05.614157 6204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:05.614286 6204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:05.623361 6204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:05.624444 6204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:05.625695 6204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:05.638041 6210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:05.638075 6212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:05.638326 6209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:05.638880 6204 server_base.cc:1034] running on GCE node
I20260926 03:00:05.639370 6204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:05.640161 6204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:05.641342 6204 hybrid_clock.cc:648] HybridClock initialized: now 1790391605641297 us; error 36 us; skew 500 ppm
I20260926 03:00:05.641594 6204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:05 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:05.644436 6204 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:00:05.645764 6204 webserver.cc:460] Webserver started at http://127.0.97.2:38947/ using document root <none> and password file <none>
I20260926 03:00:05.646386 6204 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:05.646538 6204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:05.646900 6204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:05.649685 6204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "e0c54966377c432cb519ca987cb65192"
format_stamp: "Formatted at 2026-09-26 03:00:05 on dist-test-slave-fccw"
server_key: "2fa7b6b5340ce90ca5747b4c3dfd7aaa"
server_key_iv: "aed17e7c6b3322998376142b510e8aa9"
server_key_version: "encryptionkey@0"
I20260926 03:00:05.650408 6204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e0c54966377c432cb519ca987cb65192"
format_stamp: "Formatted at 2026-09-26 03:00:05 on dist-test-slave-fccw"
server_key: "2fa7b6b5340ce90ca5747b4c3dfd7aaa"
server_key_iv: "aed17e7c6b3322998376142b510e8aa9"
server_key_version: "encryptionkey@0"
I20260926 03:00:05.655282 6204 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:00:05.658432 6219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:05.659626 6204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:00:05.659780 6204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "e0c54966377c432cb519ca987cb65192"
format_stamp: "Formatted at 2026-09-26 03:00:05 on dist-test-slave-fccw"
server_key: "2fa7b6b5340ce90ca5747b4c3dfd7aaa"
server_key_iv: "aed17e7c6b3322998376142b510e8aa9"
server_key_version: "encryptionkey@0"
I20260926 03:00:05.659986 6204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:05.684451 6204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:05.688282 6204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:05.688566 6204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:05.689621 6204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:05.691002 6204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:05.691071 6204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:05.691234 6204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:05.691296 6204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:05.723325 6204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42309
I20260926 03:00:05.723356 6331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42309 every 8 connection(s)
I20260926 03:00:05.724596 6204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:05.733410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6204
I20260926 03:00:05.733677 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:00:05.734284 388 external_mini_cluster.cc:1428] Setting key 058d9c9f1e26c3268f5e516617d75080
Sep 26 03:00:05 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391605, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:05.744601 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:05.727086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43213 (local address 127.0.97.62:34977)
0926 03:00:05.727252 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:05.727257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:05.728373 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 03:00:05.730079 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:05.730083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:05.730165 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:05.730252 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:05.731621 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.732306 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.733928 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.734578 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.734705 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:05.738278 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:05.738287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:05.738294 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:05.738335 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:05.741778 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:05.742447 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:05.742452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:05.742454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:05.742572 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:05.743008 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:05.743013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:05.743017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:05.743250 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:05.743395 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:05.744209 (+ 814us) server_negotiation.cc:299] Negotiation successful
0926 03:00:05.744361 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:00:05.746096 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:05.727548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.2:43213)
0926 03:00:05.728210 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:05.728252 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:05.729832 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:05.730440 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:05.730447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:05.730922 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:05.731482 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:05.731497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.732417 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.732421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:05.733784 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:05.733809 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:05.734763 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:05.734768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:05.735061 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:05.735729 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:05.735757 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:05.738074 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:05.741915 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:05.741921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:05.741932 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:05.742314 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:05.742689 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:05.742695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:05.742698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:05.742862 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:05.743371 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:05.743378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:05.743817 (+ 439us) client_negotiation.cc:769] Sending connection context
0926 03:00:05.744102 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 03:00:05.744646 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
I20260926 03:00:05.747474 6332 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34977
I20260926 03:00:05.747756 6332 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:05.748397 6332 heartbeater.cc:511] Master 127.0.97.62:34977 requested a full tablet report, sending...
I20260926 03:00:05.749871 6009 ts_manager.cc:194] Registered new tserver with Master: e0c54966377c432cb519ca987cb65192 (127.0.97.2:42309)
I20260926 03:00:05.750655 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43213
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:00:05.779588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:06.092423 6337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:06.092715 6337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:06.092885 6337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:06.099942 6337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:06.100085 6337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:06.100268 6337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:06.109843 6337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34977
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:06.111128 6337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:06.112458 6337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:06.125216 6342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:06.125240 6343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:06.125303 6345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:06.126143 6337 server_base.cc:1034] running on GCE node
I20260926 03:00:06.126672 6337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:06.127494 6337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:06.128683 6337 hybrid_clock.cc:648] HybridClock initialized: now 1790391606128631 us; error 54 us; skew 500 ppm
I20260926 03:00:06.128973 6337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.131996 6337 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:00:06.133385 6337 webserver.cc:460] Webserver started at http://127.0.97.3:34231/ using document root <none> and password file <none>
I20260926 03:00:06.133975 6337 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:06.134078 6337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:06.134315 6337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:06.137496 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "a0629e2b53ba48d5924c52aead31e22c"
format_stamp: "Formatted at 2026-09-26 03:00:06 on dist-test-slave-fccw"
server_key: "203e26f3b5d932f851d60e7c7d326862"
server_key_iv: "f679af32ea0109128a8f48b90a3df7ab"
server_key_version: "encryptionkey@0"
I20260926 03:00:06.138183 6337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a0629e2b53ba48d5924c52aead31e22c"
format_stamp: "Formatted at 2026-09-26 03:00:06 on dist-test-slave-fccw"
server_key: "203e26f3b5d932f851d60e7c7d326862"
server_key_iv: "f679af32ea0109128a8f48b90a3df7ab"
server_key_version: "encryptionkey@0"
I20260926 03:00:06.142774 6337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:00:06.145869 6352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:06.147125 6337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:06.147280 6337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "a0629e2b53ba48d5924c52aead31e22c"
format_stamp: "Formatted at 2026-09-26 03:00:06 on dist-test-slave-fccw"
server_key: "203e26f3b5d932f851d60e7c7d326862"
server_key_iv: "f679af32ea0109128a8f48b90a3df7ab"
server_key_version: "encryptionkey@0"
I20260926 03:00:06.147464 6337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:06.160266 6337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:06.164302 6337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:06.164569 6337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:06.165485 6337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:06.166797 6337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:06.166867 6337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:06.167018 6337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:06.167120 6337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:06.198973 6337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41065
I20260926 03:00:06.199015 6464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41065 every 8 connection(s)
I20260926 03:00:06.200189 6337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:06.207006 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6337
I20260926 03:00:06.207159 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:00:06.207602 388 external_mini_cluster.cc:1428] Setting key 0a140cd99ff318d27bfc245657184248
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:06.219130 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.202555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48251 (local address 127.0.97.62:34977)
0926 03:00:06.202719 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.202724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.203805 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 03:00:06.205606 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.205611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.205697 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.205805 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.207252 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.208175 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.210262 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.210816 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.210902 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.214097 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:06.214102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:06.214105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:06.214133 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:06.216604 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.217349 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.217353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.217355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.217477 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.217816 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.217819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.217821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.218049 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:06.218162 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:06.218733 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 03:00:06.218876 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:00:06.220069 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.202998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34977 (local address 127.0.97.3:48251)
0926 03:00:06.203660 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:06.203695 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:06.205344 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:06.206029 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:06.206037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:06.206509 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:06.207053 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:06.207099 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.208356 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.208364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:06.210131 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:06.210142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.211005 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.211010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:06.211212 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.211769 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:06.211789 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:06.213877 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:06.216759 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:06.216768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:06.216785 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:06.217251 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:06.217563 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:06.217566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:06.217568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:06.217725 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:06.218156 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:06.218162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:06.218434 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 03:00:06.218638 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 03:00:06.218964 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":268,"threads_started":1}
I20260926 03:00:06.221457 6465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34977
I20260926 03:00:06.221755 6465 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:06.222532 6465 heartbeater.cc:511] Master 127.0.97.62:34977 requested a full tablet report, sending...
I20260926 03:00:06.223810 6009 ts_manager.cc:194] Registered new tserver with Master: a0629e2b53ba48d5924c52aead31e22c (127.0.97.3:41065)
I20260926 03:00:06.224498 6009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48251
I20260926 03:00:06.234819 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/test-admin.keytab.
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.264362 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 03:00:06.269294 6472 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:06.266767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.0.97.1:35947)
0926 03:00:06.267352 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.267364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.267395 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:00:06.267631 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.267638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.267844 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.268009 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.268779 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48850: BlockingRecv error: recv got EOF from 127.0.0.1:48850 (error 108)
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":280,"threads_started":1}
W20260926 03:00:06.269573 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc)
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.270984 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.272267 388 init.cc:295] Successfully reacquired a new kerberos TGT
I20260926 03:00:06.278059 6198 heartbeater.cc:503] Master 127.0.97.62:34977 was elected leader, sending a full tablet report...
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:00:06.286928 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.273903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48858 (local address 127.0.97.1:35947)
0926 03:00:06.274092 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.274096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.274176 (+ 80us) server_negotiation.cc:407] Connection header received
0926 03:00:06.274291 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.274294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.274365 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.274442 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.275348 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.276269 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.277505 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.278080 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.278217 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.281796 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:06.281806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:06.281819 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:06.281847 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:06.284315 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.284942 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.284959 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.284961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.285145 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.285512 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.285518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.285521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.285841 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:06.286131 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:06.286438 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:00:06.286679 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:00:06.292279 6472 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:06.290650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.0.97.1:35947)
0926 03:00:06.290840 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.290845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.291037 (+ 192us) server_negotiation.cc:407] Connection header received
0926 03:00:06.291214 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.291219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.291278 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.291368 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.292029 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48868: BlockingRecv error: recv got EOF from 127.0.0.1:48868 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 03:00:06.292517 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.293983 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.295214 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:00:06.307925 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.296260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.0.97.1:35947)
0926 03:00:06.296458 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.296464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.296711 (+ 247us) server_negotiation.cc:407] Connection header received
0926 03:00:06.296880 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.296883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.296929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.297064 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.297893 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.298538 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.299728 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.300168 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.300310 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.303596 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:06.303601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:06.303604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:06.303643 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:06.305907 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.306451 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.306455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.306457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.306566 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.306846 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.306849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.306851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.307067 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:06.307255 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:06.307441 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 03:00:06.307609 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 03:00:06.312016 6472 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:06.310443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.0.97.1:35947)
0926 03:00:06.310635 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.310640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.310837 (+ 197us) server_negotiation.cc:407] Connection header received
0926 03:00:06.311027 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.311032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.311112 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.311207 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.311844 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48886: BlockingRecv error: recv got EOF from 127.0.0.1:48886 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 03:00:06.312338 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790391557890803-388-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.313954 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:06.315208 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:00:06.328431 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.316327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.0.97.1:35947)
0926 03:00:06.316504 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.316511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.316755 (+ 244us) server_negotiation.cc:407] Connection header received
0926 03:00:06.316929 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.316933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.317078 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.317162 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.318027 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.318726 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.319891 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.320307 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.320431 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.323707 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:06.323714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:06.323718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:06.323755 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:06.325962 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.326598 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.326604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.326608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.326750 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.327105 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.327112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.327115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.327360 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:06.327657 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:06.327944 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:00:06.328168 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:00:06.329902 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6070
I20260926 03:00:06.340934 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.330529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48902 (local address 127.0.97.1:35947)
0926 03:00:06.330689 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.330696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.330977 (+ 281us) server_negotiation.cc:407] Connection header received
0926 03:00:06.331141 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.331151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.331198 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.331286 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.332090 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.332710 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.333925 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.334344 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.334474 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.336279 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:06.336283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:06.336287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:06.336323 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:06.338545 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.339163 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.339174 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.339178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.339304 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.339647 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.339653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.339657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.339909 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:06.340177 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:06.340471 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 03:00:06.340659 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:00:06.341533 6193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:06.422402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6070
I20260926 03:00:06.441308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6204
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:00:06.455310 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.442125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.0.97.2:42309)
0926 03:00:06.442688 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.442695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.442719 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:00:06.442774 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.442781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.442984 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.443119 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.443980 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.444969 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.446159 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.446709 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.446817 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.450311 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:06.450321 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:06.450336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:06.450362 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:06.452957 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.453662 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.453668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.453670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.453778 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.454091 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.454094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.454096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.454379 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:06.454612 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:06.454862 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 03:00:06.455038 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":292,"threads_started":1}
I20260926 03:00:06.456489 6327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:06.538008 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6204
I20260926 03:00:06.553381 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6337
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[5964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:00:06.568599 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:06.554308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.0.97.3:41065)
0926 03:00:06.554916 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:06.554923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:06.554951 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:00:06.555007 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:06.555013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:06.555233 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:06.555406 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:06.556429 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.557526 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.558763 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:06.559331 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:06.559475 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:06.563436 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:06.563447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:06.563461 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:06.563492 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:06.566289 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.566995 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.567001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.567004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.567126 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:06.567439 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:06.567442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:06.567444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:06.567748 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:06.567970 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:06.568144 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 03:00:06.568320 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":337,"threads_started":1}
I20260926 03:00:06.569893 6460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:06.651854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6337
I20260926 03:00:06.667591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 5977
I20260926 03:00:06.668754 6039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:06.768338 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 5977
2026-09-26T03:00:06Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3827 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[6492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[6492](info): set up 2 sockets
Sep 26 03:00:06 dist-test-slave-fccw krb5kdc[6492](info): commencing operation
krb5kdc: starting...
W20260926 03:00:07.837775 388 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:00:07 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 03:00:08.030479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42493
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:08.338431 6505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:08.338759 6505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:08.338922 6505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:08.345994 6505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:08.346131 6505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:08.346246 6505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:08.346340 6505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:08.346432 6505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:08.356009 6505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:08.357335 6505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:08.358626 6505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:08.370440 6513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:08.370477 6511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:08.370366 6510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:08.371487 6505 server_base.cc:1034] running on GCE node
I20260926 03:00:08.372213 6505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:08.373680 6505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:08.374877 6505 hybrid_clock.cc:648] HybridClock initialized: now 1790391608374839 us; error 24 us; skew 500 ppm
I20260926 03:00:08.375166 6505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:08 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:08.378229 6505 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 03:00:08.379617 6505 webserver.cc:460] Webserver started at http://127.0.97.62:34089/ using document root <none> and password file <none>
I20260926 03:00:08.380318 6505 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:08.380422 6505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:08.380673 6505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:08.383723 6505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "a80c50d5c0c445a6bf3709ee0eecbb26"
format_stamp: "Formatted at 2026-09-26 03:00:08 on dist-test-slave-fccw"
server_key: "40eb37cbf431838221117be498522506"
server_key_iv: "a29adb9dd5b720331020653bf6d371e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:08.384443 6505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "a80c50d5c0c445a6bf3709ee0eecbb26"
format_stamp: "Formatted at 2026-09-26 03:00:08 on dist-test-slave-fccw"
server_key: "40eb37cbf431838221117be498522506"
server_key_iv: "a29adb9dd5b720331020653bf6d371e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:08.389853 6505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260926 03:00:08.393280 6520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:08.394599 6505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:00:08.394781 6505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "a80c50d5c0c445a6bf3709ee0eecbb26"
format_stamp: "Formatted at 2026-09-26 03:00:08 on dist-test-slave-fccw"
server_key: "40eb37cbf431838221117be498522506"
server_key_iv: "a29adb9dd5b720331020653bf6d371e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:08.394999 6505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:08.420015 6505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:08.424278 6505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:08.424561 6505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:08.441752 6505 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42493
I20260926 03:00:08.441772 6571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42493 every 8 connection(s)
I20260926 03:00:08.443017 6505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:08.446525 6572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:08.448201 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6505
I20260926 03:00:08.448401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:08.448984 388 external_mini_cluster.cc:1428] Setting key 6ac11de1de1ba9a80b3b51ceb2780f2c
I20260926 03:00:08.458606 6572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: Bootstrap starting.
I20260926 03:00:08.462419 6572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:08.463591 6572 log.cc:824] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:00:08 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:08.466415 6572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: No bootstrap required, opened a new log
I20260926 03:00:08.471482 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:08.450924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.97.62:42493)
0926 03:00:08.451717 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:08.451728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:08.451769 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:00:08.453844 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:08.453865 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:08.454236 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:08.454535 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:08.456228 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:08.457606 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:08.459253 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:08.459889 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:08.460032 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:08.464586 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:08.464598 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:08.464612 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:08.464649 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:08.467663 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:08.468334 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:08.468340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:08.468345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:08.468502 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:08.468866 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:08.468872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:08.468875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:08.469439 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:08.469739 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:08.470163 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 03:00:08.470551 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":337,"threads_started":1}
I20260926 03:00:08.474548 6572 raft_consensus.cc:348] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } }
I20260926 03:00:08.474891 6572 raft_consensus.cc:374] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:08.474993 6572 raft_consensus.cc:729] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80c50d5c0c445a6bf3709ee0eecbb26, State: Initialized, Role: FOLLOWER
I20260926 03:00:08.475499 6572 consensus_queue.cc:260] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } }
I20260926 03:00:08.475728 6572 raft_consensus.cc:388] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:08.475863 6572 raft_consensus.cc:482] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:08.476035 6572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:08.478106 6572 raft_consensus.cc:504] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } }
I20260926 03:00:08.478631 6572 leader_election.cc:302] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a80c50d5c0c445a6bf3709ee0eecbb26; no voters:
I20260926 03:00:08.479164 6572 leader_election.cc:288] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:08.479280 6577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:08.480531 6577 raft_consensus.cc:686] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 1 LEADER]: Becoming Leader. State: Replica: a80c50d5c0c445a6bf3709ee0eecbb26, State: Running, Role: LEADER
I20260926 03:00:08.481105 6577 consensus_queue.cc:237] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } }
I20260926 03:00:08.481328 6572 sys_catalog.cc:564] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:08.483711 6579 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a80c50d5c0c445a6bf3709ee0eecbb26. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } }
I20260926 03:00:08.484104 6579 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:08.483860 6578 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } }
I20260926 03:00:08.484200 6578 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:08.489988 6584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:08.494278 6584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:08.502065 6584 catalog_manager.cc:1347] Generated new cluster ID: 610be077267b48729512193805fe9b5c
I20260926 03:00:08.502177 6584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:08.514599 6584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:08.515797 6584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:08.524596 6584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: Generated new TSK 0
I20260926 03:00:08.525414 6584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 03:00:08.578003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42493
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:08.893272 6598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:08.893561 6598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:08.893752 6598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:08.901525 6598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:08.901649 6598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:08.901829 6598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:08.911108 6598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:08.912371 6598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:08.913699 6598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:08.926756 6606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:08.926868 6603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:08.926882 6604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:08.927788 6598 server_base.cc:1034] running on GCE node
I20260926 03:00:08.928357 6598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:08.929240 6598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:08.930467 6598 hybrid_clock.cc:648] HybridClock initialized: now 1790391608930421 us; error 55 us; skew 500 ppm
I20260926 03:00:08.930766 6598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:08 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:08.933704 6598 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:00:08.935096 6598 webserver.cc:460] Webserver started at http://127.0.97.1:36981/ using document root <none> and password file <none>
I20260926 03:00:08.935748 6598 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:08.935895 6598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:08.936180 6598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:08.939251 6598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "fd226c00ab9141e998c7d13428d6b268"
format_stamp: "Formatted at 2026-09-26 03:00:08 on dist-test-slave-fccw"
server_key: "aa8344fc063ee30b142b812c8e5eed92"
server_key_iv: "f25badae1845cf77ead55d5ae1148dbf"
server_key_version: "encryptionkey@0"
I20260926 03:00:08.940111 6598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fd226c00ab9141e998c7d13428d6b268"
format_stamp: "Formatted at 2026-09-26 03:00:08 on dist-test-slave-fccw"
server_key: "aa8344fc063ee30b142b812c8e5eed92"
server_key_iv: "f25badae1845cf77ead55d5ae1148dbf"
server_key_version: "encryptionkey@0"
I20260926 03:00:08.944923 6598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260926 03:00:08.948066 6613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:08.949422 6598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:00:08.949594 6598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "fd226c00ab9141e998c7d13428d6b268"
format_stamp: "Formatted at 2026-09-26 03:00:08 on dist-test-slave-fccw"
server_key: "aa8344fc063ee30b142b812c8e5eed92"
server_key_iv: "f25badae1845cf77ead55d5ae1148dbf"
server_key_version: "encryptionkey@0"
I20260926 03:00:08.949807 6598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:08.964630 6598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:08.968752 6598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:08.969090 6598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:08.969967 6598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:08.971264 6598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:08.971374 6598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:08.971513 6598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:08.971580 6598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:09.005679 6598 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39395
I20260926 03:00:09.005723 6725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39395 every 8 connection(s)
I20260926 03:00:09.006970 6598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:09.016525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6598
I20260926 03:00:09.016733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:00:09.017494 388 external_mini_cluster.cc:1428] Setting key 80a96ed62c14c9213e01ab06a474c7b8
Sep 26 03:00:09 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:09.027748 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:09.009592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38179 (local address 127.0.97.62:42493)
0926 03:00:09.009970 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:09.009975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:09.011014 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 03:00:09.012725 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:09.012730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:09.012816 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:09.012930 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:09.014533 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.015286 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.017250 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.017704 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.017795 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:09.021406 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:09.021411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:09.021415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:09.021443 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:09.024173 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:09.025176 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:09.025180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:09.025182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:09.025298 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:09.025798 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:09.025804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:09.025809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:09.026166 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:09.026314 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:09.027035 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 03:00:09.027274 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":190,"threads_started":1}
I20260926 03:00:09.028275 6729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:09.010166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.1:38179)
0926 03:00:09.010852 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:09.010889 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:09.012473 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:09.013218 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:09.013226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:09.013708 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:09.014252 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:09.014268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.015441 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.015444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:09.017120 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:09.017136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.017914 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.017921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:09.018173 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:09.018799 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:09.018819 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:09.021209 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:09.024345 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:09.024353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:09.024367 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:09.025046 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:09.025410 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:09.025416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:09.025422 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:09.025673 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:09.026284 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:09.026289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:09.026699 (+ 410us) client_negotiation.cc:769] Sending connection context
0926 03:00:09.026900 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 03:00:09.027227 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":273,"threads_started":1}
I20260926 03:00:09.029722 6726 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42493
I20260926 03:00:09.030016 6726 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:09.030740 6726 heartbeater.cc:511] Master 127.0.97.62:42493 requested a full tablet report, sending...
I20260926 03:00:09.032616 6537 ts_manager.cc:194] Registered new tserver with Master: fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395)
I20260926 03:00:09.034051 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38179
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 03:00:09.064049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42493
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:09.376888 6732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:09.377156 6732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:09.377247 6732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:09.384407 6732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:09.384498 6732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:09.384632 6732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:09.393784 6732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:09.394883 6732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:09.396095 6732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:09.408599 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:09.408630 6740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:09.408607 6737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:09.409736 6732 server_base.cc:1034] running on GCE node
I20260926 03:00:09.410334 6732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:09.411211 6732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:09.412431 6732 hybrid_clock.cc:648] HybridClock initialized: now 1790391609412384 us; error 58 us; skew 500 ppm
I20260926 03:00:09.412725 6732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:09 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:09.415915 6732 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:00:09.417367 6732 webserver.cc:460] Webserver started at http://127.0.97.2:43593/ using document root <none> and password file <none>
I20260926 03:00:09.418026 6732 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:09.418162 6732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:09.418442 6732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:09.421303 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd03a6db3c494566b954e4a9bc73d704"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "d22eba1042078f5f550dcd377ec009de"
server_key_iv: "f5de473f83440f4552b7c69f5fd293f4"
server_key_version: "encryptionkey@0"
I20260926 03:00:09.422046 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd03a6db3c494566b954e4a9bc73d704"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "d22eba1042078f5f550dcd377ec009de"
server_key_iv: "f5de473f83440f4552b7c69f5fd293f4"
server_key_version: "encryptionkey@0"
I20260926 03:00:09.426759 6732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 03:00:09.429952 6747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:09.431082 6732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:00:09.431242 6732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "bd03a6db3c494566b954e4a9bc73d704"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "d22eba1042078f5f550dcd377ec009de"
server_key_iv: "f5de473f83440f4552b7c69f5fd293f4"
server_key_version: "encryptionkey@0"
I20260926 03:00:09.431417 6732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:09.458531 6732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:09.462205 6732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:09.462499 6732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:09.463382 6732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:09.464627 6732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:09.464713 6732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:09.464888 6732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:09.464939 6732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:09.496004 6732 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44699
I20260926 03:00:09.496050 6859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44699 every 8 connection(s)
I20260926 03:00:09.497328 6732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:09.502324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6732
I20260926 03:00:09.502485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:00:09.502900 388 external_mini_cluster.cc:1428] Setting key f804903a682da5757f27e71d54ea23f4
Sep 26 03:00:09 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:09.517427 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:09.499751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57725 (local address 127.0.97.62:42493)
0926 03:00:09.499992 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:09.499999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:09.501199 (+ 1200us) server_negotiation.cc:407] Connection header received
0926 03:00:09.503017 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:09.503023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:09.503123 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:09.503230 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:09.504763 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.505838 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.507304 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.508006 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.508146 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:09.511927 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:09.511931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:09.511934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:09.511962 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:09.514442 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:09.515127 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:09.515133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:09.515142 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:09.515256 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:09.515738 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:09.515745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:09.515750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:09.516019 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:09.516169 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:09.516790 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 03:00:09.517054 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:00:09.518381 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:09.500329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:57725)
0926 03:00:09.500988 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:09.501094 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:09.502764 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:09.503461 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:09.503473 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:09.504030 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:09.504590 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:09.504613 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.505971 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.505975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:09.507160 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:09.507171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.508215 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.508223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:09.508515 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:09.509271 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:09.509301 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:09.511719 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:09.514579 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:09.514587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:09.514603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:09.515006 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:09.515385 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:09.515390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:09.515393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:09.515641 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:09.516144 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:09.516151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:09.516461 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 03:00:09.516713 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 03:00:09.517270 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":246,"threads_started":1}
I20260926 03:00:09.519843 6860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42493
I20260926 03:00:09.520195 6860 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:09.521106 6860 heartbeater.cc:511] Master 127.0.97.62:42493 requested a full tablet report, sending...
I20260926 03:00:09.522624 6537 ts_manager.cc:194] Registered new tserver with Master: bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699)
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:00:09.523456 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:00:09.548538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42493
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:09.856856 6865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:09.857144 6865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:09.857342 6865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:09.864264 6865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:09.864382 6865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:09.864557 6865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:09.873626 6865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:09.874791 6865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:09.876070 6865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:09.888434 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:09.888583 6873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:09.888463 6871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:09.889242 6865 server_base.cc:1034] running on GCE node
I20260926 03:00:09.889819 6865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:09.890781 6865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:09.891997 6865 hybrid_clock.cc:648] HybridClock initialized: now 1790391609891942 us; error 45 us; skew 500 ppm
I20260926 03:00:09.892315 6865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:09 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:09.900352 6865 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:00:09.901844 6865 webserver.cc:460] Webserver started at http://127.0.97.3:35561/ using document root <none> and password file <none>
I20260926 03:00:09.902494 6865 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:09.902635 6865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:09.902922 6865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:09.905718 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "5cb395bdd48542e1baa56277b11090f2"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "283b102280f7bf69e1c3267a3785ebf5"
server_key_iv: "7a0eba4b0e97400860cba8a27b168faa"
server_key_version: "encryptionkey@0"
I20260926 03:00:09.906471 6865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5cb395bdd48542e1baa56277b11090f2"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "283b102280f7bf69e1c3267a3785ebf5"
server_key_iv: "7a0eba4b0e97400860cba8a27b168faa"
server_key_version: "encryptionkey@0"
I20260926 03:00:09.911368 6865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:00:09.914564 6880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:09.915827 6865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:00:09.915997 6865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "5cb395bdd48542e1baa56277b11090f2"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "283b102280f7bf69e1c3267a3785ebf5"
server_key_iv: "7a0eba4b0e97400860cba8a27b168faa"
server_key_version: "encryptionkey@0"
I20260926 03:00:09.916181 6865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:09.944497 6865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:09.948840 6865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:09.949193 6865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:09.950104 6865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:09.951400 6865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:09.951499 6865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:09.951638 6865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:09.951730 6865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:09.983493 6865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44007
I20260926 03:00:09.983537 6992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44007 every 8 connection(s)
I20260926 03:00:09.984731 6865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:09.986024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6865
I20260926 03:00:09.986191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:00:09.986713 388 external_mini_cluster.cc:1428] Setting key 02113a08aadd9543cbe90c501dafc1df
Sep 26 03:00:09 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:10.005625 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:09.987455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52401 (local address 127.0.97.62:42493)
0926 03:00:09.987630 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:09.987636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:09.989041 (+ 1405us) server_negotiation.cc:407] Connection header received
0926 03:00:09.991002 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:09.991007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:09.991086 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:09.991194 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:09.992572 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.993362 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.994830 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.995239 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.995362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:09.999532 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:09.999540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:09.999545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:09.999581 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.002738 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.003435 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.003443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.003446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.003568 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.003998 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.004003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.004007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.004267 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.004421 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.005087 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.005286 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:00:10.006484 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:09.988053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.3:52401)
0926 03:00:09.988878 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:09.988920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:09.990770 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:09.991414 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:09.991421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:09.991889 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:09.992411 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:09.992428 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.993501 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.993507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:09.994692 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:09.994703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:09.995482 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:09.995489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:09.995710 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:09.996308 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:09.996359 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:09.999330 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:10.002869 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.002878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.002893 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.003318 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.003690 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.003695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.003699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.003878 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.004391 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:10.004399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:10.004705 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 03:00:10.004952 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 03:00:10.005482 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":301,"threads_started":1}
I20260926 03:00:10.007771 6993 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42493
I20260926 03:00:10.008071 6993 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:10.008836 6993 heartbeater.cc:511] Master 127.0.97.62:42493 requested a full tablet report, sending...
I20260926 03:00:10.010262 6537 ts_manager.cc:194] Registered new tserver with Master: 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:10.011018 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52401
I20260926 03:00:10.014024 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
I20260926 03:00:10.036761 6726 heartbeater.cc:503] Master 127.0.97.62:42493 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260926 03:00:10.060045 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34195
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42493,127.0.97.61:34195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:10.372565 6998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:10.372848 6998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:10.372936 6998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:10.379993 6998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:10.380090 6998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:10.380136 6998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:10.380185 6998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:10.380228 6998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:10.389994 6998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42493,127.0.97.61:34195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
W20260926 03:00:10.390275 6998 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:42493,127.0.97.61:34195'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 03:00:10.391203 6998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:10.392438 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:10.403707 7003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:10.403666 7004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:10.403760 7006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:10.404479 6998 server_base.cc:1034] running on GCE node
I20260926 03:00:10.405233 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:10.406476 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:10.407678 6998 hybrid_clock.cc:648] HybridClock initialized: now 1790391610407604 us; error 47 us; skew 500 ppm
I20260926 03:00:10.407986 6998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:10 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:10.411129 6998 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 03:00:10.412403 6998 webserver.cc:460] Webserver started at http://127.0.97.61:40355/ using document root <none> and password file <none>
I20260926 03:00:10.413050 6998 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:10.413146 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:10.413401 6998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:10.416136 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/instance:
uuid: "ae770f4b38954985b59301605d8d61f5"
format_stamp: "Formatted at 2026-09-26 03:00:10 on dist-test-slave-fccw"
server_key: "b46c47bcb7834239ac1eb913b1115592"
server_key_iv: "0aa01cf1d1635bf45354f787392f9c9a"
server_key_version: "encryptionkey@0"
I20260926 03:00:10.416829 6998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal/instance:
uuid: "ae770f4b38954985b59301605d8d61f5"
format_stamp: "Formatted at 2026-09-26 03:00:10 on dist-test-slave-fccw"
server_key: "b46c47bcb7834239ac1eb913b1115592"
server_key_iv: "0aa01cf1d1635bf45354f787392f9c9a"
server_key_version: "encryptionkey@0"
I20260926 03:00:10.421839 6998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:00:10.424894 7013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:10.426131 6998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:00:10.426291 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
uuid: "ae770f4b38954985b59301605d8d61f5"
format_stamp: "Formatted at 2026-09-26 03:00:10 on dist-test-slave-fccw"
server_key: "b46c47bcb7834239ac1eb913b1115592"
server_key_iv: "0aa01cf1d1635bf45354f787392f9c9a"
server_key_version: "encryptionkey@0"
I20260926 03:00:10.426501 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:10.448540 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:10.452080 6998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:10.452325 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:10.469614 6998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34195
I20260926 03:00:10.469630 7064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34195 every 8 connection(s)
I20260926 03:00:10.470891 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:00:10.474346 7065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:10.478323 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 6998
I20260926 03:00:10.481823 7065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:00:10 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:00:10 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:10.501035 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.485614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40336 (local address 127.0.97.62:42493)
0926 03:00:10.485829 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:10.485835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:10.486511 (+ 676us) server_negotiation.cc:407] Connection header received
0926 03:00:10.486703 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:10.486721 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:10.486794 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:10.486899 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:10.488090 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.488853 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.490297 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.490701 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.490796 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.494500 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:10.494504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:10.494507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:10.494537 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.497698 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.498516 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.498521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.498524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.498658 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.499055 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.499060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.499063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.499418 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.499575 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.500463 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.500654 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:00:10.501727 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.480533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.0.97.61:34195)
0926 03:00:10.481363 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:10.481376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:10.481418 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:00:10.483098 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:10.483119 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:10.483489 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:10.483853 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:10.485505 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.487221 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.488648 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.489437 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.489634 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.494058 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:10.494068 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:10.494082 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:10.494120 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.497169 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.497783 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.497794 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.497802 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.497999 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.498398 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.498404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.498407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.498969 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.499247 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.500193 (+ 946us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.500473 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":429,"threads_started":1}
I20260926 03:00:10.501771 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.485616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:40336)
0926 03:00:10.486365 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:10.486418 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:10.486578 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:10.486949 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:10.486953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:10.487661 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:10.487952 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.487963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.488989 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.489012 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.490177 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.490187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.490918 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.490928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.491116 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.491690 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:10.491727 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:10.494336 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:10.497821 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.497828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.497834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.498321 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.498778 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.498783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.498786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.498951 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.499548 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:10.499557 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:10.499778 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 03:00:10.500059 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 03:00:10.500638 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":144,"thread_start_us":387,"threads_started":1}
I20260926 03:00:10.503651 7065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:00:10 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:00:10.515329 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.504391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:34900)
0926 03:00:10.504564 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:10.504581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:10.504686 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:10.504935 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:10.504939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:10.505284 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:10.505399 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.505406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.506332 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.506340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.507197 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.507206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.507895 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.507900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.508126 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.508647 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:10.508662 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:10.510561 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:10.513191 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.513197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.513200 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.513548 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.513871 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.513875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.513878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.514002 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.514445 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:10.514449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:10.514581 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 03:00:10.514706 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 03:00:10.514985 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:00:10.515404 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.504574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.0.97.61:34195)
0926 03:00:10.504696 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:10.504700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:10.504718 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:00:10.504766 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:10.504773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:10.504830 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:10.504921 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:10.505534 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.506216 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.507338 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.507754 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.507879 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.510725 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:10.510730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:10.510733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:10.510757 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.513068 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.513650 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.513657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.513661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.513766 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.514105 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.514109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.514112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.514339 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.514477 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.514793 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.515106 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":27}
I20260926 03:00:10.520273 7065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Bootstrap starting.
I20260926 03:00:10.523427 7065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:10.524477 7065 log.cc:824] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:10.526028 6860 heartbeater.cc:503] Master 127.0.97.62:42493 was elected leader, sending a full tablet report...
I20260926 03:00:10.527086 7065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: No bootstrap required, opened a new log
I20260926 03:00:10.534291 7065 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } }
I20260926 03:00:10.534621 7065 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:10.534720 7065 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae770f4b38954985b59301605d8d61f5, State: Initialized, Role: FOLLOWER
I20260926 03:00:10.535327 7065 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } }
I20260926 03:00:10.536110 7071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } } }
I20260926 03:00:10.536413 7071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:00:10.537213 7065 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:00:10.545660 7082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:00:10.545845 7082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:00:10.547303 6998 master_runner.cc:428] Detected that this master ae770f4b38954985b59301605d8d61f5 is joining an existing cluster
I20260926 03:00:10.547406 6998 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:34195
I20260926 03:00:10.570937 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.550962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.0.97.61:34195)
0926 03:00:10.551185 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:10.551191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:10.554370 (+ 3179us) server_negotiation.cc:407] Connection header received
0926 03:00:10.554551 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:10.554555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:10.554613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:10.554702 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:10.556015 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.557214 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.559589 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.560346 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.560458 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.563691 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:10.563699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:10.563707 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:10.563744 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.567570 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.568529 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.568537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.568542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.568679 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.569144 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.569151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.569158 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.569536 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.569681 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.570207 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.570443 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":55}
I20260926 03:00:10.571035 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.551473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.0.97.62:42493)
0926 03:00:10.551611 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:10.551615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:10.553635 (+ 2020us) server_negotiation.cc:407] Connection header received
0926 03:00:10.553815 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:10.553821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:10.553910 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:10.554017 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:10.556401 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.557234 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.559511 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.560163 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.560258 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.563668 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:10.563676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:10.563681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:10.563717 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.567644 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.568575 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.568581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.568584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.568753 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.569263 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.569268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.569271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.569645 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.569761 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.570412 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.570659 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 03:00:10.572079 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.550924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:40350)
0926 03:00:10.553537 (+ 2613us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:10.553554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:10.553698 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:10.554193 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:10.554198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:10.555903 (+ 1705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:10.556253 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.556261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.557440 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.557445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.559347 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.559361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.560546 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.560552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.560742 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.561666 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:10.561690 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:10.563445 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:10.567823 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.567828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.567832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.568429 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.568907 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.568911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.568915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.569130 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.569860 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:10.569865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:10.570066 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 03:00:10.571278 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0926 03:00:10.571601 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2402,"mutex_wait_us":121,"spinlock_wait_cycles":21888,"thread_start_us":2282,"threads_started":1}
I20260926 03:00:10.572573 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.551230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:34908)
0926 03:00:10.554282 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:10.554296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:10.554429 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:10.554714 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:10.554718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:10.555442 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:10.555856 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.555870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.557434 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.557441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.559453 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.559468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.560764 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.560772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.561023 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.561638 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:10.561712 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:10.563512 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:10.567718 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.567728 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.567735 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.568418 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.568809 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.568813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.568818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.569040 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.569767 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:10.569774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:10.569946 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 03:00:10.571621 (+ 1675us) client_negotiation.cc:240] Negotiation successful
0926 03:00:10.571960 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2960,"mutex_wait_us":107,"spinlock_wait_cycles":18816,"thread_start_us":2690,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 26 03:00:10 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:00:10.597191 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.578804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.97.61:34195)
0926 03:00:10.578991 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:10.578998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:10.579622 (+ 624us) server_negotiation.cc:407] Connection header received
0926 03:00:10.579916 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:10.579922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:10.580046 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:10.580152 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:10.581315 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.582398 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.586234 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.586673 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.586849 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.590448 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:10.590466 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:10.590472 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:10.590508 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.594012 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.594900 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.594906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.594910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.595059 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.595510 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.595515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.595518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.595855 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.596000 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.596565 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.596789 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:00:10.597416 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.578787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:34922)
0926 03:00:10.579470 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:10.579528 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:10.579702 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:10.580191 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:10.580196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:10.580818 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:10.581125 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.581150 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.582535 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.585082 (+ 2547us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.586065 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.586076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.586885 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.586890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.587108 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.587732 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:10.587749 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:10.590235 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:10.594153 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.594167 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.594177 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.594755 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.595181 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.595189 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.595195 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.595371 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.596000 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:10.596012 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:10.596229 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 03:00:10.596438 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 03:00:10.596808 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":305,"threads_started":1}
I20260926 03:00:10.600016 6537 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:34195: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ae770f4b38954985b59301605d8d61f5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 34195
}
attrs {
promote: true
}
}
dest_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26"
cas_config_opid_index: -1
I20260926 03:00:10.601207 6537 consensus_queue.cc:237] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } }
I20260926 03:00:10.604496 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ae770f4b38954985b59301605d8d61f5 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } }
I20260926 03:00:10.605898 7040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260926 03:00:10.609887 7094 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:34195
I20260926 03:00:10.610292 7094 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } }
I20260926 03:00:10.610509 7094 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:10.610575 6998 master.cc:537] Master@127.0.97.61:34195 shutting down...
I20260926 03:00:10.610739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40409
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:00:10.617664 7040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Refusing update from remote peer a80c50d5c0c445a6bf3709ee0eecbb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 03:00:10.623694 7095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:10.625680 6998 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:10.626113 6998 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:10.626338 6998 tablet_replica.cc:330] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: stopping tablet replica
W20260926 03:00:10.627269 6523 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 03:00:10.633806 6998 master.cc:552] Master@127.0.97.61:34195 shutdown complete.
I20260926 03:00:10.633932 6998 master_runner.cc:305] Clearing existing system tablet
I20260926 03:00:10.634454 6998 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:10.634599 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:10.638373 7091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.637881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:34926)
0926 03:00:10.638075 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:10.638180 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 03:00:10.638768 6523 consensus_peers.cc:487] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 -> Peer ae770f4b38954985b59301605d8d61f5 (127.0.97.61:34195): Couldn't send request to peer ae770f4b38954985b59301605d8d61f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:00:10.642026 6998 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:00:10.649569 7100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:10.649976 6998 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:10.650216 6998 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.004s
I20260926 03:00:10.650328 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
uuid: "ae770f4b38954985b59301605d8d61f5"
format_stamp: "Formatted at 2026-09-26 03:00:10 on dist-test-slave-fccw"
server_key: "b46c47bcb7834239ac1eb913b1115592"
server_key_iv: "0aa01cf1d1635bf45354f787392f9c9a"
server_key_version: "encryptionkey@0"
I20260926 03:00:10.655226 6998 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:10.661991 6998 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:00:10.662154 6998 log.cc:1196] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 03:00:10.662560 6998 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Deleting consensus metadata
I20260926 03:00:10.662801 6998 master_runner.cc:315] Copying system tablet from 127.0.97.62:42493
I20260926 03:00:10.665243 6998 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:42493
I20260926 03:00:10.679790 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.666015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.0.97.62:42493)
0926 03:00:10.666205 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:10.666211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:10.666709 (+ 498us) server_negotiation.cc:407] Connection header received
0926 03:00:10.666891 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:10.666897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:10.666993 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:10.667098 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:10.668099 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.669130 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.670848 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.671416 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.671535 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.673718 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:10.673723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:10.673728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:10.673762 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:10.677018 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.677721 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.677726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.677729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.677860 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:10.678269 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:10.678273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:10.678276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:10.678619 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:10.678764 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:10.679195 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 03:00:10.679420 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:00:10.679841 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:10.665937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:40354)
0926 03:00:10.666586 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:10.666605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:10.666745 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:10.667142 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:10.667147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:10.667664 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:10.667927 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.667944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.669275 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.669281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.670706 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:10.670720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:10.671572 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:10.671578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:10.671753 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:10.672475 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:10.672498 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:10.673519 (+ 1021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:10.677139 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.677145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.677149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.677605 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.677988 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:10.677993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:10.677996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:10.678158 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:10.678738 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:10.678743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:10.678911 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 03:00:10.679080 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 03:00:10.679356 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":400,"threads_started":1}
I20260926 03:00:10.681110 6557 tablet_copy_service.cc:140] P a80c50d5c0c445a6bf3709ee0eecbb26: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ae770f4b38954985b59301605d8d61f5 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40354)
I20260926 03:00:10.681341 6557 tablet_copy_service.cc:161] P a80c50d5c0c445a6bf3709ee0eecbb26: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ae770f4b38954985b59301605d8d61f5 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40354: session id = ae770f4b38954985b59301605d8d61f5-00000000000000000000000000000000
I20260926 03:00:10.685928 6557 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:00:10.688138 6998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:00:10.694670 6998 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: tablet copy: Starting download of 0 data blocks...
I20260926 03:00:10.694995 6998 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: tablet copy: Starting download of 1 WAL segments...
I20260926 03:00:10.697664 6998 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:00:10.701930 6557 tablet_copy_service.cc:342] P a80c50d5c0c445a6bf3709ee0eecbb26: Request end of tablet copy session ae770f4b38954985b59301605d8d61f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40354
I20260926 03:00:10.702157 6557 tablet_copy_service.cc:433] P a80c50d5c0c445a6bf3709ee0eecbb26: ending tablet copy session ae770f4b38954985b59301605d8d61f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ae770f4b38954985b59301605d8d61f5
I20260926 03:00:10.713809 6998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:10.716455 7107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:10.719714 7108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:10.727151 7110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:10.727787 6998 server_base.cc:1034] running on GCE node
I20260926 03:00:10.728144 6998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:10.728794 6998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:10.730053 6998 hybrid_clock.cc:648] HybridClock initialized: now 1790391610729992 us; error 33 us; skew 500 ppm
I20260926 03:00:10.730345 6998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:10 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:10.732542 6998 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260926 03:00:10.733834 6998 webserver.cc:460] Webserver started at http://127.0.97.61:46121/ using document root <none> and password file <none>
I20260926 03:00:10.734246 6998 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:10.734368 6998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:10.737591 6998 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 03:00:10.739598 7117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:10.740037 6998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:00:10.740219 6998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
uuid: "ae770f4b38954985b59301605d8d61f5"
format_stamp: "Formatted at 2026-09-26 03:00:10 on dist-test-slave-fccw"
server_key: "b46c47bcb7834239ac1eb913b1115592"
server_key_iv: "0aa01cf1d1635bf45354f787392f9c9a"
server_key_version: "encryptionkey@0"
I20260926 03:00:10.740444 6998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:10.758991 6998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:10.763000 6998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:10.785295 6998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34195
I20260926 03:00:10.785328 7168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34195 every 8 connection(s)
I20260926 03:00:10.786386 6998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:00:10.792672 7169 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:00:10.796238 7169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Bootstrap starting.
I20260926 03:00:10.811708 7169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:00:10.812191 7169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: Bootstrap complete.
I20260926 03:00:10.813244 7169 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } }
I20260926 03:00:10.813503 7169 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae770f4b38954985b59301605d8d61f5, State: Initialized, Role: LEARNER
I20260926 03:00:10.813771 7169 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } }
I20260926 03:00:10.814599 7172 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } }
I20260926 03:00:10.814883 7169 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:10.814992 7172 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:00:10.822631 7183 catalog_manager.cc:1259] Loaded cluster ID: 610be077267b48729512193805fe9b5c
I20260926 03:00:10.822758 7183 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: loading cluster ID for follower catalog manager: success
I20260926 03:00:10.826352 7183 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: acquiring CA information for follower catalog manager: success
I20260926 03:00:10.829169 7183 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:00:10.982106 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:10.982486 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:10.982650 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:10.990639 7097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:10.990749 7097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:10.990850 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:10.990963 7097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:10.991050 7097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:11.002871 7097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:11.004215 7097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:11.006315 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:00:11.013643 6993 heartbeater.cc:503] Master 127.0.97.62:42493 was elected leader, sending a full tablet report...
W20260926 03:00:11.018810 7188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:11.019309 7189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:11.019620 7191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:11.020403 7097 server_base.cc:1034] running on GCE node
I20260926 03:00:11.021185 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:11.022532 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:11.023753 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790391611023695 us; error 55 us; skew 500 ppm
I20260926 03:00:11.024075 7097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:11.027021 7097 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 03:00:11.028499 7097 webserver.cc:460] Webserver started at http://127.0.97.60:44155/ using document root <none> and password file <none>
I20260926 03:00:11.029234 7097 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:11.029378 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:11.029649 7097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:11.032373 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/instance:
uuid: "bb9850d450f64613a023c5b3e841da85"
format_stamp: "Formatted at 2026-09-26 03:00:11 on dist-test-slave-fccw"
server_key: "f04bca83fbd67fb22eaa0e20d78995fb"
server_key_iv: "62f8462f0df57cc2a7634e088ea532e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:11.033183 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal/instance:
uuid: "bb9850d450f64613a023c5b3e841da85"
format_stamp: "Formatted at 2026-09-26 03:00:11 on dist-test-slave-fccw"
server_key: "f04bca83fbd67fb22eaa0e20d78995fb"
server_key_iv: "62f8462f0df57cc2a7634e088ea532e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:11.038147 7097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:00:11.041939 7198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:11.043336 7097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:00:11.043529 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
uuid: "bb9850d450f64613a023c5b3e841da85"
format_stamp: "Formatted at 2026-09-26 03:00:11 on dist-test-slave-fccw"
server_key: "f04bca83fbd67fb22eaa0e20d78995fb"
server_key_iv: "62f8462f0df57cc2a7634e088ea532e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:11.043725 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:11.070595 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:11.074496 7097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:11.074774 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:11.096742 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40409
I20260926 03:00:11.096791 7249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40409 every 8 connection(s)
I20260926 03:00:11.098734 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:00:11.103453 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:11.107483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 7097
I20260926 03:00:11.110652 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:11.130432 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.114349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.0.97.62:42493)
0926 03:00:11.114540 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.114546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.115020 (+ 474us) server_negotiation.cc:407] Connection header received
0926 03:00:11.115408 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.115414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.115492 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.115595 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.117349 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.118135 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.120103 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.120534 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.120670 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.124804 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.124809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.124812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.124844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.127791 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.128485 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.128489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.128491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.128601 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.129055 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.129059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.129062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.129409 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.129537 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.130057 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.130210 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:00:11.131382 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.109709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.0.97.60:40409)
0926 03:00:11.110840 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.110852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.110895 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:00:11.112484 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.112505 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.112928 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.113342 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.115121 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.117173 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.118639 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.119655 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.119847 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.124271 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.124281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.124295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.124340 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.126955 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.127660 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.127668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.127674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.127806 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.128187 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.128192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.128196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.128885 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.129154 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.129618 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.130014 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":936,"spinlock_wait_cycles":31488,"thread_start_us":745,"threads_started":1}
I20260926 03:00:11.131419 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.114217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:40358)
0926 03:00:11.114890 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.114934 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.115127 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.115651 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.115657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.116684 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.117157 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.117172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.118280 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.118288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.119937 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.119949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.120742 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.120758 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.120912 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.121529 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.121549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.124644 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.127909 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.127919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.127926 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.128379 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.128708 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.128712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.128715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.128921 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.129524 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.129530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.129686 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.129941 (+ 255us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.130199 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":1,"thread_start_us":333,"threads_started":1}
I20260926 03:00:11.133049 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:00:11.146034 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.133658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:34938)
0926 03:00:11.133841 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.133858 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.133947 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.134697 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.134700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.134972 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.135087 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.135093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.136320 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.136323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.137261 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.137271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.137887 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.137891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.138037 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.138475 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.138489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.140773 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.143760 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.143765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.143769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.144115 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.144478 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.144483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.144487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.144612 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.145146 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.145152 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.145290 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.145423 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.145700 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:00:11.146059 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.133838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.97.61:34195)
0926 03:00:11.134437 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.134441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.134456 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:00:11.134501 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.134504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.134579 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.134679 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.135198 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.136204 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.137375 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.137779 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.137859 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.140969 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.140974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.140977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.141049 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.143623 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.144228 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.144235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.144238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.144372 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.144721 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.144726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.144730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.144968 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.145190 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.145499 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.145700 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":317,"threads_started":1}
I20260926 03:00:11.147064 7250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 03:00:11.158154 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.147634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.0.1:55388)
0926 03:00:11.147754 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.147784 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.147853 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.148129 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.148131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.148474 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.148599 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.148607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.149668 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.149671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.150443 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.150452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.151003 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.151007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.151149 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.151590 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.151605 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.153526 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.156274 (+ 2748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.156277 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.156280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.156607 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.156904 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.156907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.156909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.157102 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.157522 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.157525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.157626 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.157725 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.157890 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 03:00:11.158260 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.147809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.0.97.60:40409)
0926 03:00:11.147931 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.147935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.147948 (+ 13us) server_negotiation.cc:407] Connection header received
0926 03:00:11.147980 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.147999 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.148045 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.148112 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.148701 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.149559 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.150547 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.150891 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.150986 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.153689 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.153693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.153697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.153722 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.156170 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.156698 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.156702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.156704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.156800 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.157188 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.157191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.157193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.157431 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.157536 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.157774 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.157985 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":27}
I20260926 03:00:11.162488 7250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Bootstrap starting.
I20260926 03:00:11.165990 7250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:11.166977 7250 log.cc:824] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:11.169404 7250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: No bootstrap required, opened a new log
I20260926 03:00:11.176308 7250 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } }
I20260926 03:00:11.176592 7250 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:11.176731 7250 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9850d450f64613a023c5b3e841da85, State: Initialized, Role: FOLLOWER
I20260926 03:00:11.177357 7250 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } }
I20260926 03:00:11.178112 7257 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } } }
I20260926 03:00:11.178435 7257 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:00:11.179060 7250 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:11.186565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6598
W20260926 03:00:11.188783 7268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:00:11.188971 7268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:00:11.195043 7097 master_runner.cc:428] Detected that this master bb9850d450f64613a023c5b3e841da85 is joining an existing cluster
I20260926 03:00:11.195173 7097 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:40409
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 03:00:11.213165 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.188067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.0.97.1:39395)
0926 03:00:11.189526 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.189535 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.189564 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:00:11.189626 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.189634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.189919 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.190131 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.191928 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.193479 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.194753 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.195559 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.195717 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.202774 (+ 7057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.202792 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.202815 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.202895 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.208614 (+ 5719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.209885 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.209899 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.209905 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.210110 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.210681 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.210689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.210694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.211165 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.211485 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.212340 (+ 855us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.212628 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309,"thread_start_us":1170,"threads_started":1}
I20260926 03:00:11.216248 6721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:11.233299 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.200667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.0.97.62:42493)
0926 03:00:11.200994 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.201040 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.214457 (+ 13417us) server_negotiation.cc:407] Connection header received
0926 03:00:11.214830 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.214837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.214919 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.214992 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.216343 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.218057 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.221457 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.222628 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.222739 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.225609 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.225616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.225639 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.225685 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.229461 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.230444 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.230451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.230455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.230646 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.231150 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.231155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.231159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.231526 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.231667 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.232601 (+ 934us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.232842 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260926 03:00:11.234604 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.200242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:40364)
0926 03:00:11.211168 (+ 10926us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.211189 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.214575 (+ 3386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.215236 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.215243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.215886 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.216168 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.216177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.218196 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.218203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.220972 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.220983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.223063 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.223073 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.223290 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.224048 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.224072 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.225389 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.229671 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.229675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.229679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.230282 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.230835 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.230840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.230845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.231017 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.231757 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.231764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.232229 (+ 465us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.233503 (+ 1274us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.233918 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10777,"spinlock_wait_cycles":4480,"thread_start_us":12362,"threads_started":1}
I20260926 03:00:11.238411 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.214034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.0.97.60:40409)
0926 03:00:11.214201 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.214206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.221987 (+ 7781us) server_negotiation.cc:407] Connection header received
0926 03:00:11.222207 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.222212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.222290 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.222369 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.223927 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.225226 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.227406 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.228211 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.228325 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.231154 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.231160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.231165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.231198 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.235452 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.236166 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.236175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.236178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.236342 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.236754 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.236759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.236762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.237156 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.237286 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.237751 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.237947 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:00:11.239566 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.214201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.0.1:55392)
0926 03:00:11.221496 (+ 7295us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.221518 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.222074 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.222551 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.222556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.223359 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.223765 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.223785 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.225461 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.225465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.227242 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.227259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.228803 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.228809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.229042 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.229703 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.229741 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.230950 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.235556 (+ 4606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.235559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.235562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.236042 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.236478 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.236482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.236485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.236637 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.237364 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.237368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.237507 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.238796 (+ 1289us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.239092 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7176,"thread_start_us":7021,"threads_started":1}
I20260926 03:00:11.253125 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.201292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.0.97.61:34195)
0926 03:00:11.201481 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.201490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.212592 (+ 11102us) server_negotiation.cc:407] Connection header received
0926 03:00:11.213148 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.213155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.213255 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.213331 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.214707 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.215749 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.225886 (+ 10137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.226545 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.226647 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.245114 (+ 18467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.245123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.245130 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.245178 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.249741 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.250910 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.250918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.250923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.251072 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.251605 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.251610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.251613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.251916 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.252023 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.252590 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.252805 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:00:11.254866 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.200468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:34940)
0926 03:00:11.212494 (+ 12026us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.212512 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.212968 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.213545 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.213550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.214210 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.214539 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.214552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.222024 (+ 7472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.222029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.223739 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.223753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.239938 (+ 16185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.239952 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.242955 (+ 3003us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.243674 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.243702 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.244863 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.250002 (+ 5139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.250028 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.250035 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.250734 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.251215 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.251220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.251224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.251428 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.252104 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.252109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.252285 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.254061 (+ 1776us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.254411 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11884,"thread_start_us":13006,"threads_started":1}
I20260926 03:00:11.264196 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.234336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.97.61:34195)
0926 03:00:11.253674 (+ 19338us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.253680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.253697 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:00:11.253753 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.253758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.253849 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.253951 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:11.257553 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.258699 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.261579 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.262982 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.263080 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.263279 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.263805 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.263927 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19221,"thread_start_us":22630,"threads_started":1}
I20260926 03:00:11.265513 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.237743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:34944)
0926 03:00:11.238208 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.238224 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.238339 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.256960 (+ 18621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.256968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.257070 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:11.257352 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.257373 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.258863 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.258870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.261378 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.261395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.263451 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.263457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.263619 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.263672 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.264554 (+ 882us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.264687 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":394,"threads_started":1}
I20260926 03:00:11.266942 7144 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 03:00:11.267710 7172 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a80c50d5c0c445a6bf3709ee0eecbb26. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } }
I20260926 03:00:11.267939 7172 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: LEARNER
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391608, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 03:00:11.276240 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.241128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.0.1:55396)
0926 03:00:11.258523 (+ 17395us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.258546 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.258660 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.259243 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.259249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.259894 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.260245 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.260258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.261929 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.261935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.263506 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.263521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.264938 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.264945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.265592 (+ 647us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.266425 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.266451 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.269341 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.273537 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.273542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.273546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.274039 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.274473 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.274477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.274481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.274621 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.275298 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.275303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.275466 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.275607 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.275840 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17283,"thread_start_us":17167,"threads_started":1}
I20260926 03:00:11.276337 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.240823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.0.97.60:40409)
0926 03:00:11.240973 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.240980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.258654 (+ 17674us) server_negotiation.cc:407] Connection header received
0926 03:00:11.258803 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.258809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.258915 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.259021 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.260430 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.261718 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.263688 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.264503 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.264615 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.269556 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.269563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.269569 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.269613 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.273400 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.274160 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.274166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.274170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.274348 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.274789 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.274795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.274799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.275167 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.275324 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.275736 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.275973 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:00:11.278476 6537 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:40409: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bb9850d450f64613a023c5b3e841da85"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 40409
}
attrs {
promote: true
}
}
dest_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26"
cas_config_opid_index: 5
I20260926 03:00:11.279430 6537 consensus_queue.cc:237] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } }
I20260926 03:00:11.281265 7279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb9850d450f64613a023c5b3e841da85 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.286454 7225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:11.289672 7144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 LEARNER]: Refusing update from remote peer a80c50d5c0c445a6bf3709ee0eecbb26: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:00:11.290156 7279 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:40409
I20260926 03:00:11.290455 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 03:00:11.290479 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.290885 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:11.295953 7144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb9850d450f64613a023c5b3e841da85 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.301234 7172 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.301489 7172 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:00:11.302011 7284 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: attempting to promote NON_VOTER ae770f4b38954985b59301605d8d61f5 to VOTER
I20260926 03:00:11.302825 7284 consensus_queue.cc:237] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } }
I20260926 03:00:11.305078 7144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 LEARNER]: Refusing update from remote peer a80c50d5c0c445a6bf3709ee0eecbb26: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 03:00:11.305773 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 03:00:11.308041 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ae770f4b38954985b59301605d8d61f5 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.312548 7144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, ae770f4b38954985b59301605d8d61f5 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.313347 7283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.313279 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.313558 7283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:11.313577 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:11.317425 7172 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.317660 7172 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:00:11.325380 7097 master.cc:537] Master@127.0.97.60:40409 shutting down...
I20260926 03:00:11.325469 7225 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Refusing update from remote peer a80c50d5c0c445a6bf3709ee0eecbb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:00:11.326521 7224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Refusing update from remote peer a80c50d5c0c445a6bf3709ee0eecbb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 03:00:11.327232 7204 tablet.cc:2369] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:00:11.328181 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:00:11.335590 6523 consensus_peers.cc:487] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 -> Peer bb9850d450f64613a023c5b3e841da85 (127.0.97.60:40409): Couldn't send request to peer bb9850d450f64613a023c5b3e841da85. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 03:00:11.340143 7097 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:11.340596 7097 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:11.340818 7097 tablet_replica.cc:330] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: stopping tablet replica
I20260926 03:00:11.347760 7097 master.cc:552] Master@127.0.97.60:40409 shutdown complete.
I20260926 03:00:11.347853 7097 master_runner.cc:305] Clearing existing system tablet
I20260926 03:00:11.348313 7097 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:11.348436 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:11.351624 7097 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 03:00:11.353370 7289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:11.353773 7097 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:11.354056 7097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:00:11.354209 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
uuid: "bb9850d450f64613a023c5b3e841da85"
format_stamp: "Formatted at 2026-09-26 03:00:11 on dist-test-slave-fccw"
server_key: "f04bca83fbd67fb22eaa0e20d78995fb"
server_key_iv: "62f8462f0df57cc2a7634e088ea532e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:11.359586 7097 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:11.362402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6598
I20260926 03:00:11.366861 7097 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:00:11.367015 7097 log.cc:1196] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 03:00:11.367501 7097 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Deleting consensus metadata
I20260926 03:00:11.367771 7097 master_runner.cc:315] Copying system tablet from 127.0.97.62:42493
I20260926 03:00:11.370219 7097 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:42493
I20260926 03:00:11.383692 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39395
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42493
--webserver_port=36981
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:00:11.385911 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.371098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.0.97.62:42493)
0926 03:00:11.371321 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.371326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.371852 (+ 526us) server_negotiation.cc:407] Connection header received
0926 03:00:11.372028 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.372037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.372130 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.372255 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.373272 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.374592 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.376413 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.377221 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.377374 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.379625 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.379633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.379641 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.379685 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.383342 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.384022 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.384027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.384030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.384161 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.384536 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.384542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.384545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.384795 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.384962 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.385338 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.385537 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 03:00:11.385969 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.371056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:40376)
0926 03:00:11.371711 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.371731 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.371890 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.372297 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.372302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.372850 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.373117 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.373128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.374764 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.374769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.376254 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.376268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.377381 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.377387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.377564 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.378294 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.378315 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.379436 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.383474 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.383480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.383485 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.383873 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.384279 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.384284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.384289 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.384432 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.384916 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.384921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.385112 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.385249 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.385538 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":392,"threads_started":1}
I20260926 03:00:11.386744 6557 tablet_copy_service.cc:140] P a80c50d5c0c445a6bf3709ee0eecbb26: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bb9850d450f64613a023c5b3e841da85 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40376)
I20260926 03:00:11.386909 6557 tablet_copy_service.cc:161] P a80c50d5c0c445a6bf3709ee0eecbb26: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bb9850d450f64613a023c5b3e841da85 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40376: session id = bb9850d450f64613a023c5b3e841da85-00000000000000000000000000000000
I20260926 03:00:11.389261 6557 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:00:11.390916 7097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:00:11.395570 7097 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: tablet copy: Starting download of 0 data blocks...
I20260926 03:00:11.395866 7097 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: tablet copy: Starting download of 1 WAL segments...
I20260926 03:00:11.398001 7097 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:00:11.401624 6557 tablet_copy_service.cc:342] P a80c50d5c0c445a6bf3709ee0eecbb26: Request end of tablet copy session bb9850d450f64613a023c5b3e841da85-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:40376
I20260926 03:00:11.401751 6557 tablet_copy_service.cc:433] P a80c50d5c0c445a6bf3709ee0eecbb26: ending tablet copy session bb9850d450f64613a023c5b3e841da85-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bb9850d450f64613a023c5b3e841da85
I20260926 03:00:11.413085 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:11.415858 7297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:11.416206 7298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:11.420228 7300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:11.422168 7097 server_base.cc:1034] running on GCE node
I20260926 03:00:11.422472 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:11.423107 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:11.424304 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790391611424259 us; error 34 us; skew 500 ppm
I20260926 03:00:11.424548 7097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:11.426656 7097 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260926 03:00:11.430123 7097 webserver.cc:460] Webserver started at http://127.0.97.60:44947/ using document root <none> and password file <none>
I20260926 03:00:11.430469 7097 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:11.430568 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:11.433439 7097 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:00:11.435328 7307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:11.435820 7097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:00:11.435978 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
uuid: "bb9850d450f64613a023c5b3e841da85"
format_stamp: "Formatted at 2026-09-26 03:00:11 on dist-test-slave-fccw"
server_key: "f04bca83fbd67fb22eaa0e20d78995fb"
server_key_iv: "62f8462f0df57cc2a7634e088ea532e1"
server_key_version: "encryptionkey@0"
I20260926 03:00:11.436239 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:11.459219 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:11.463259 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:11.495217 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40409
I20260926 03:00:11.495270 7358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40409 every 8 connection(s)
I20260926 03:00:11.498011 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:00:11.505919 7359 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:00:11.509665 7359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Bootstrap starting.
I20260926 03:00:11.527140 7359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:00:11.527592 7359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Bootstrap complete.
I20260926 03:00:11.528555 7359 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } }
I20260926 03:00:11.529527 7359 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb9850d450f64613a023c5b3e841da85, State: Initialized, Role: LEARNER
I20260926 03:00:11.529834 7359 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } }
I20260926 03:00:11.530580 7362 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.530834 7359 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:11.530988 7362 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:00:11.542433 7373 catalog_manager.cc:1259] Loaded cluster ID: 610be077267b48729512193805fe9b5c
I20260926 03:00:11.542546 7373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: loading cluster ID for follower catalog manager: success
I20260926 03:00:11.544770 7373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: acquiring CA information for follower catalog manager: success
I20260926 03:00:11.546528 7373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:00:11.782015 7296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:11.782397 7296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:11.782567 7296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:11.793380 7296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:11.793573 7296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:11.793776 7296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:11.805828 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.784548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.0.1:55402)
0926 03:00:11.800637 (+ 16089us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.800668 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.800850 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.801275 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.801283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.801310 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:11.801512 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.801521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.802779 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.802786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.803911 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.803923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.805055 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.805061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.805218 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.805263 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.805370 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.805520 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15974,"thread_start_us":15893,"threads_started":1}
I20260926 03:00:11.805863 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.784716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.0.97.60:40409)
0926 03:00:11.800189 (+ 15473us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.800199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.800773 (+ 574us) server_negotiation.cc:407] Connection header received
0926 03:00:11.801023 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.801031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.801119 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.801241 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:11.801657 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.802637 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.804063 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.804855 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.805034 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.805274 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.805389 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.805519 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15288,"thread_start_us":15478,"threads_started":1}
I20260926 03:00:11.806988 7334 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260926 03:00:11.807574 7362 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a80c50d5c0c445a6bf3709ee0eecbb26. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:11.808307 7362 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:00:11.809307 7296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36981
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:11.810577 7296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:11.811944 7296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:11.825522 7381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:11.825497 7383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:11.825558 7380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:11.830488 7296 server_base.cc:1034] running on GCE node
I20260926 03:00:11.831527 7296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:11.832396 7296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:11.833638 7296 hybrid_clock.cc:648] HybridClock initialized: now 1790391611833603 us; error 48 us; skew 500 ppm
I20260926 03:00:11.833993 7296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:11.837386 7296 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:00:11.838783 7296 webserver.cc:460] Webserver started at http://127.0.97.1:36981/ using document root <none> and password file <none>
I20260926 03:00:11.839464 7296 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:11.839613 7296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:11.844803 7296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:11.847316 7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:11.848585 7296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:00:11.848750 7296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "fd226c00ab9141e998c7d13428d6b268"
format_stamp: "Formatted at 2026-09-26 03:00:08 on dist-test-slave-fccw"
server_key: "aa8344fc063ee30b142b812c8e5eed92"
server_key_iv: "f25badae1845cf77ead55d5ae1148dbf"
server_key_version: "encryptionkey@0"
I20260926 03:00:11.849709 7296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:11.864240 7296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:11.868325 7296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:11.868641 7296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:11.869537 7296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:11.870790 7296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:11.870901 7296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:11.871026 7296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:11.871109 7296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:11.905221 7296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39395
I20260926 03:00:11.905288 7502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39395 every 8 connection(s)
I20260926 03:00:11.906618 7296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:11.908402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 7296
I20260926 03:00:11.908730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6732
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:00:11 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391611, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:11.965790 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.937145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60945 (local address 127.0.97.60:40409)
0926 03:00:11.937350 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.937355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.946384 (+ 9029us) server_negotiation.cc:407] Connection header received
0926 03:00:11.946518 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.946523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.946597 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.946694 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.948938 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.950011 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.952176 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.952908 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.953139 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.959001 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.959005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.959008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.959037 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.962315 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.963142 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.963146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.963149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.963284 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.964302 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.964306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.964309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.964634 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.964757 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.965248 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.965446 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 03:00:11.966926 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.910336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.97.1:42183)
0926 03:00:11.936221 (+ 25885us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.936273 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.939364 (+ 3091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.946542 (+ 7178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.946546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.948485 (+ 1939us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.948764 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.948776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.950314 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.950321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.952210 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.952220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.953480 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.953486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.953723 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.954669 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.954700 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.957862 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.962193 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.962202 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.962219 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.962903 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.963343 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.963350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.963356 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.963592 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.964164 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.964171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.964596 (+ 425us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.964880 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.965425 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25573,"thread_start_us":25286,"threads_started":1}
I20260926 03:00:11.967574 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.911582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.0.97.2:44699)
0926 03:00:11.932151 (+ 20569us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.932164 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.933084 (+ 920us) server_negotiation.cc:407] Connection header received
0926 03:00:11.933401 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.933414 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.933843 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.934008 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.942449 (+ 8441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.944505 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.949533 (+ 5028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.950656 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.950758 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.958772 (+ 8014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.958809 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.958828 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.958883 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.962923 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.963788 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.963798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.963803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.963937 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.964779 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.964787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.964791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.965233 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.965826 (+ 593us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.966943 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.967190 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20339,"thread_start_us":20603,"threads_started":1}
I20260926 03:00:11.967686 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.910639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42819 (local address 127.0.97.62:42493)
0926 03:00:11.940191 (+ 29552us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.940200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.940879 (+ 679us) server_negotiation.cc:407] Connection header received
0926 03:00:11.941039 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.941044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.941143 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.941286 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.945399 (+ 4113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.946829 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.953626 (+ 6797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.954392 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.954513 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.959589 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.959598 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.959604 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.959644 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.963424 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.964271 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.964277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.964283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.964864 (+ 581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.965379 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.965384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.965387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.965722 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.965855 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.967145 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.967334 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29395,"thread_start_us":29272,"threads_started":1}
I20260926 03:00:11.967765 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.911586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42183 (local address 127.0.97.61:34195)
0926 03:00:11.945323 (+ 33737us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:11.945331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:11.945363 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:00:11.945444 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:11.945452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:11.945558 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:11.945702 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:11.949083 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.950153 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.952341 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.952943 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.953087 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.958132 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:11.958141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:11.958149 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:11.958193 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:11.962027 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.963049 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.963057 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.963064 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.963208 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:11.963702 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:11.963709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:11.963715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:11.964025 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:11.964174 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:11.967310 (+ 3136us) server_negotiation.cc:299] Negotiation successful
0926 03:00:11.967456 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33589,"thread_start_us":33504,"threads_started":1}
I20260926 03:00:11.966955 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.910336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.1:42819)
0926 03:00:11.940748 (+ 30412us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.940770 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.940909 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.942743 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.942759 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.943830 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.944792 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.944822 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.947060 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.947070 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.953480 (+ 6410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.953491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.954836 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.954845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.955030 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.956099 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.956127 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.958786 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.963685 (+ 4899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.963689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.963692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.964171 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.965101 (+ 930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.965106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.965109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.965256 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.965924 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.965928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.966063 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.966200 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.966476 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30218,"mutex_wait_us":420,"thread_start_us":29545,"threads_started":1}
I20260926 03:00:11.967056 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:11.937499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.97.1:60945)
0926 03:00:11.946278 (+ 8779us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:11.946295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:11.946409 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:11.946859 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:11.946865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:11.947475 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:11.948306 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.948319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.950182 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.950187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.952025 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:11.952038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:11.954124 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:11.954131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:11.954310 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:11.954880 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:11.954903 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:11.958084 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:11.962469 (+ 4385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.962474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.962477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.963025 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.963409 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:11.963416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:11.963419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:11.963571 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:11.964843 (+ 1272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:11.964850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:11.964961 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 03:00:11.966256 (+ 1295us) client_negotiation.cc:240] Negotiation successful
0926 03:00:11.966637 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8657,"mutex_wait_us":113,"thread_start_us":8531,"threads_started":1}
I20260926 03:00:11.969202 6855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:11.970199 7504 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42493
I20260926 03:00:11.970364 7503 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40409
I20260926 03:00:11.970536 7503 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:11.970541 7504 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:11.971024 7505 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34195
I20260926 03:00:11.971184 7505 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:11.971490 7503 heartbeater.cc:511] Master 127.0.97.60:40409 requested a full tablet report, sending...
I20260926 03:00:11.971796 7505 heartbeater.cc:511] Master 127.0.97.61:34195 requested a full tablet report, sending...
I20260926 03:00:11.972471 7504 heartbeater.cc:511] Master 127.0.97.62:42493 requested a full tablet report, sending...
I20260926 03:00:11.973815 6537 ts_manager.cc:194] Re-registered known tserver with Master: fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395)
I20260926 03:00:11.973949 7134 ts_manager.cc:194] Registered new tserver with Master: fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395)
I20260926 03:00:11.975047 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42819
I20260926 03:00:11.975792 7324 ts_manager.cc:194] Registered new tserver with Master: fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395)
I20260926 03:00:12.068769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6732
I20260926 03:00:12.084780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44699
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42493
--webserver_port=43593
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:00:12.250865 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: true } } }
I20260926 03:00:12.251302 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:12.252842 7279 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: attempting to promote NON_VOTER bb9850d450f64613a023c5b3e841da85 to VOTER
I20260926 03:00:12.253549 7279 consensus_queue.cc:237] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } }
I20260926 03:00:12.255309 7334 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 LEARNER]: Refusing update from remote peer a80c50d5c0c445a6bf3709ee0eecbb26: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:00:12.255391 7144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Refusing update from remote peer a80c50d5c0c445a6bf3709ee0eecbb26: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 03:00:12.255832 7279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:00:12.256215 7516 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 03:00:12.259611 7144 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb9850d450f64613a023c5b3e841da85 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.258916 7279 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb9850d450f64613a023c5b3e841da85 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.266641 7520 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.266999 7520 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:00:12.267519 7334 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bb9850d450f64613a023c5b3e841da85 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.268728 7516 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.268796 7517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.269148 7516 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:12.269200 7517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:12.269567 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.269819 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:12.271128 7362 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:12.271360 7362 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:00:12.404902 7515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:12.405197 7515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:12.405287 7515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:12.412115 7515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:12.412205 7515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:12.412317 7515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:12.422041 7515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43593
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:12.423211 7515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:12.424356 7515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:12.437172 7525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:12.437238 7526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:12.437206 7528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:12.438488 7515 server_base.cc:1034] running on GCE node
I20260926 03:00:12.439087 7515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:12.439990 7515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:12.441174 7515 hybrid_clock.cc:648] HybridClock initialized: now 1790391612441135 us; error 49 us; skew 500 ppm
I20260926 03:00:12.441501 7515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:12 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:12.444649 7515 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:00:12.446106 7515 webserver.cc:460] Webserver started at http://127.0.97.2:43593/ using document root <none> and password file <none>
I20260926 03:00:12.446748 7515 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:12.446890 7515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:12.451747 7515 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:00:12.454566 7535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:12.455848 7515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:00:12.455998 7515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "bd03a6db3c494566b954e4a9bc73d704"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "d22eba1042078f5f550dcd377ec009de"
server_key_iv: "f5de473f83440f4552b7c69f5fd293f4"
server_key_version: "encryptionkey@0"
I20260926 03:00:12.456910 7515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:12.482203 7515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:12.486347 7515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:12.486573 7515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:12.487437 7515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:12.488850 7515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:12.488934 7515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:12.489063 7515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:12.489123 7515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 03:00:12.524392 7515 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44699
I20260926 03:00:12.524482 7647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44699 every 8 connection(s)
I20260926 03:00:12.525821 7515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:12.533497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 7515
I20260926 03:00:12.533810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6865
Sep 26 03:00:12 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 03:00:12 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 26 03:00:12 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 26 03:00:12 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391612, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:00:12.568490 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:12.528857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48329 (local address 127.0.97.62:42493)
0926 03:00:12.548927 (+ 20070us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:12.548935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:12.548959 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:00:12.549050 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:12.549058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:12.549153 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:12.550648 (+ 1495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:12.550832 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.552269 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.554501 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.555559 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.556314 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:12.560605 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:12.560611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:12.560615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:12.560653 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:12.565034 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.565822 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.565829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.565834 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.565997 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.566470 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.566476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.566481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.566840 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:12.566992 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:12.567826 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 03:00:12.568048 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19909,"thread_start_us":26333,"threads_started":1}
I20260926 03:00:12.570039 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:12.529151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:48329)
0926 03:00:12.541446 (+ 12295us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:12.541511 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:12.544846 (+ 3335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:12.549493 (+ 4647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:12.549500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:12.550096 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:12.550467 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:12.550481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.552464 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.552473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:12.554291 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:12.554302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.555781 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.555787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:12.556005 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:12.556763 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:12.556783 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:12.560142 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:12.565149 (+ 5007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:12.565158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:12.565177 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:12.565641 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:12.566111 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:12.566117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:12.566122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:12.566350 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:12.566960 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:12.566972 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:12.567394 (+ 422us) client_negotiation.cc:769] Sending connection context
0926 03:00:12.567692 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 03:00:12.568601 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11889,"mutex_wait_us":2,"thread_start_us":15743,"threads_started":1}
I20260926 03:00:12.570681 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:12.546621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39603 (local address 127.0.97.61:34195)
0926 03:00:12.547177 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:12.547183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:12.554043 (+ 6860us) server_negotiation.cc:407] Connection header received
0926 03:00:12.554267 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:12.554272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:12.554368 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:12.554446 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:12.556159 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.557158 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.558877 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.559455 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.559556 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:12.564571 (+ 5015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:12.564578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:12.564584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:12.564661 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:12.567705 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.568564 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.568571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.568576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.568709 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.569173 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.569180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.569186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.569486 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:12.569618 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:12.570124 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 03:00:12.570332 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":347,"threads_started":1}
I20260926 03:00:12.570811 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:12.535473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.97.3:44007)
0926 03:00:12.547745 (+ 12272us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:12.547756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:12.548367 (+ 611us) server_negotiation.cc:407] Connection header received
0926 03:00:12.548566 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:12.548574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:12.548818 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:12.551380 (+ 2562us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:12.551591 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.553199 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.555520 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.556809 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.556936 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:12.562782 (+ 5846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:12.562796 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:12.562813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:12.562861 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:12.567447 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.568355 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.568365 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.568369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.568521 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.568928 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.568933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.568935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.569341 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:12.569596 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:12.569981 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 03:00:12.570254 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12068,"thread_start_us":11868,"threads_started":1}
I20260926 03:00:12.572098 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:12.547172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.97.2:39603)
0926 03:00:12.553948 (+ 6776us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:12.553966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:12.554132 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:12.554670 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:12.554675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:12.555519 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:12.555985 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:12.555997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.557287 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.557294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:12.558720 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:12.558734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.559849 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.559857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:12.560107 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:12.561174 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:12.561196 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:12.564347 (+ 3151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:12.567850 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:12.567854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:12.567857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:12.568424 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:12.568845 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:12.568852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:12.568859 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:12.569054 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:12.569678 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:12.569682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:12.569818 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 03:00:12.571433 (+ 1615us) client_negotiation.cc:240] Negotiation successful
0926 03:00:12.571697 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6651,"mutex_wait_us":60,"thread_start_us":6353,"threads_started":1}
I20260926 03:00:12.572256 6988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:12.572753 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:12.528599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32807 (local address 127.0.97.60:40409)
0926 03:00:12.544150 (+ 15551us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:12.544159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:12.546456 (+ 2297us) server_negotiation.cc:407] Connection header received
0926 03:00:12.546688 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:12.546696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:12.547254 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:12.550236 (+ 2982us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:12.550430 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.551557 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.554091 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.554727 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.556284 (+ 1557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:12.561212 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:12.561218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:12.561223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:12.561258 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:12.566034 (+ 4776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.570688 (+ 4654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.570693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.570697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.570857 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:12.571329 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:12.571333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:12.571336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:12.571656 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:12.571813 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:12.572173 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:00:12.572376 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15399,"thread_start_us":15276,"threads_started":1}
I20260926 03:00:12.573172 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:12.529151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.97.2:32807)
0926 03:00:12.546365 (+ 17214us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:12.546384 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:12.546509 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:12.547708 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:12.547724 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:12.548696 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:12.550029 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:12.550060 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.551742 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.551747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:12.553900 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:12.553914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:12.555474 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:12.555481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:12.555861 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:12.556625 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:12.556659 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:12.560952 (+ 4293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:12.566561 (+ 5609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:12.566565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:12.566569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:12.568438 (+ 1869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:12.571003 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:12.571009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:12.571014 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:12.571167 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:12.571781 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:12.571785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:12.571927 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 03:00:12.572079 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 03:00:12.572348 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17058,"mutex_wait_us":65,"spinlock_wait_cycles":29312,"thread_start_us":16797,"threads_started":1}
I20260926 03:00:12.573424 7649 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42493
I20260926 03:00:12.573736 7649 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:12.573946 7650 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34195
I20260926 03:00:12.574052 7650 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:12.574653 7648 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40409
I20260926 03:00:12.574697 7649 heartbeater.cc:511] Master 127.0.97.62:42493 requested a full tablet report, sending...
I20260926 03:00:12.574784 7648 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:12.575188 7648 heartbeater.cc:511] Master 127.0.97.60:40409 requested a full tablet report, sending...
I20260926 03:00:12.575544 7650 heartbeater.cc:511] Master 127.0.97.61:34195 requested a full tablet report, sending...
I20260926 03:00:12.576416 7324 ts_manager.cc:194] Registered new tserver with Master: bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699)
I20260926 03:00:12.576484 6537 ts_manager.cc:194] Re-registered known tserver with Master: bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699)
I20260926 03:00:12.576889 7134 ts_manager.cc:194] Registered new tserver with Master: bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699)
I20260926 03:00:12.577534 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48329
I20260926 03:00:12.657698 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6865
I20260926 03:00:12.673743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44007
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42493
--webserver_port=35561
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:00:12.977763 7504 heartbeater.cc:503] Master 127.0.97.62:42493 was elected leader, sending a full tablet report...
W20260926 03:00:12.994333 7661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:12.994575 7661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:12.994663 7661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:13.001850 7661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:13.001986 7661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:13.002152 7661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:13.012522 7661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35561
--tserver_master_addrs=127.0.97.62:42493,127.0.97.61:34195,127.0.97.60:40409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:13.013803 7661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:13.015002 7661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:13.029518 7667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:13.029585 7670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:13.029552 7668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:13.030586 7661 server_base.cc:1034] running on GCE node
I20260926 03:00:13.031186 7661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:13.032121 7661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:13.033331 7661 hybrid_clock.cc:648] HybridClock initialized: now 1790391613033294 us; error 29 us; skew 500 ppm
I20260926 03:00:13.033648 7661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:13 dist-test-slave-fccw krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:13.036761 7661 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:00:13.038229 7661 webserver.cc:460] Webserver started at http://127.0.97.3:35561/ using document root <none> and password file <none>
I20260926 03:00:13.038870 7661 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:13.039019 7661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:13.044059 7661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 03:00:13.046650 7677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:13.047866 7661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:00:13.048027 7661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "5cb395bdd48542e1baa56277b11090f2"
format_stamp: "Formatted at 2026-09-26 03:00:09 on dist-test-slave-fccw"
server_key: "283b102280f7bf69e1c3267a3785ebf5"
server_key_iv: "7a0eba4b0e97400860cba8a27b168faa"
server_key_version: "encryptionkey@0"
I20260926 03:00:13.048960 7661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:13.061828 7661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:13.067031 7661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:13.067361 7661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:13.068528 7661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:13.070253 7661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:13.070363 7661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:13.070470 7661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:13.070556 7661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:13.106573 7661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44007
I20260926 03:00:13.106658 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44007 every 8 connection(s)
I20260926 03:00:13.107966 7661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:13.112519 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 7661
Sep 26 03:00:13 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 26 03:00:13 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260926 03:00:13.144836 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.110893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44665 (local address 127.0.97.62:42493)
0926 03:00:13.117879 (+ 6986us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.117886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.120108 (+ 2222us) server_negotiation.cc:407] Connection header received
0926 03:00:13.122042 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.122047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.122145 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.122265 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.124703 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.125861 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.128547 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.129196 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.129312 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.137288 (+ 7976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.137295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.137299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.137342 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.141153 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.142037 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.142042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.142045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.142172 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.142716 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.142720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.142723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.143031 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.143166 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.144068 (+ 902us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.144359 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6848,"thread_start_us":6791,"threads_started":1}
I20260926 03:00:13.146117 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.112145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.3:44665)
0926 03:00:13.120007 (+ 7862us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.120029 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.121835 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.122511 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.122528 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.123621 (+ 1093us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:13.124542 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.124570 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.125996 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.126004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.128425 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.128442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.132134 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.132148 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.132563 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.133418 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:13.133458 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:13.137086 (+ 3628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:13.141298 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:13.141310 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:13.141331 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:13.141923 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:13.142312 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:13.142319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:13.142325 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:13.142612 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:13.143149 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:13.143159 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:13.143603 (+ 444us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.143947 (+ 344us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.144531 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7699,"mutex_wait_us":62,"thread_start_us":14921,"threads_started":1}
I20260926 03:00:13.147997 7791 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42493
I20260926 03:00:13.148386 7791 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:13.149973 7791 heartbeater.cc:511] Master 127.0.97.62:42493 requested a full tablet report, sending...
Sep 26 03:00:13 dist-test-slave-fccw krb5kdc[6492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391613, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260926 03:00:13.151911 6537 ts_manager.cc:194] Re-registered known tserver with Master: 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:13.153327 6537 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44665
I20260926 03:00:13.161769 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.119346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.97.3:35115)
0926 03:00:13.127335 (+ 7989us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.127357 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.127490 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.131593 (+ 4103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.131600 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.132289 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:13.133221 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.135227 (+ 2006us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.138328 (+ 3101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.138336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.140793 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.140812 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.148383 (+ 7571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.148391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.148676 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.149391 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:13.149433 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:13.152827 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:13.158599 (+ 5772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:13.158607 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:13.158614 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:13.159283 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:13.159942 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:13.159951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:13.159956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:13.160123 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:13.160881 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:13.160886 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:13.161072 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.161218 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.161500 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7843,"mutex_wait_us":3,"thread_start_us":711,"threads_started":1}
I20260926 03:00:13.163007 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.114887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35115 (local address 127.0.97.60:40409)
0926 03:00:13.131061 (+ 16174us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.131069 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.131094 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:00:13.131166 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.131173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.131259 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.131387 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.135509 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.138143 (+ 2634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.141052 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.147340 (+ 6288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.148109 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.153118 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.153128 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.153136 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.153187 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.158397 (+ 5210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.159473 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.159481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.159488 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.159785 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.160251 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.160257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.160262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.160670 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.162433 (+ 1763us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.162550 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.162773 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16021,"thread_start_us":20407,"threads_started":1}
I20260926 03:00:13.165293 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.111589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.97.3:44705)
0926 03:00:13.114210 (+ 2621us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.114277 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.121760 (+ 7483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.123564 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.123568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.123924 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:13.124863 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.124873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.128931 (+ 4058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.128936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.130073 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.130090 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.138773 (+ 8683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.138779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.139373 (+ 594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.140009 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:13.140033 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:13.142715 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:13.160920 (+ 18205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:13.160928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:13.160934 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:13.162196 (+ 1262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:13.163313 (+ 1117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:13.163317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:13.163319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:13.163449 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:13.164280 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:13.164284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:13.164444 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.164583 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.164884 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2240,"mutex_wait_us":580,"thread_start_us":1927,"threads_started":1}
I20260926 03:00:13.165599 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.110794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44705 (local address 127.0.97.61:34195)
0926 03:00:13.121211 (+ 10417us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.121230 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.121254 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:00:13.123128 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.123134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.123241 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.123339 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.125100 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.128784 (+ 3684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.137103 (+ 8319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.137766 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.139712 (+ 1946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.142918 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.142924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.142928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.142961 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.158442 (+ 15481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.162319 (+ 3877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.162324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.162327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.162488 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.163573 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.163579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.163582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.163945 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.164123 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.164758 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.165071 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10295,"thread_start_us":10149,"threads_started":1}
I20260926 03:00:13.168715 7790 heartbeater.cc:347] Connected to a master server at 127.0.97.60:40409
I20260926 03:00:13.168736 7792 heartbeater.cc:347] Connected to a master server at 127.0.97.61:34195
I20260926 03:00:13.168879 7792 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:13.169811 7792 heartbeater.cc:511] Master 127.0.97.61:34195 requested a full tablet report, sending...
I20260926 03:00:13.168882 7790 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:13.170836 7790 heartbeater.cc:511] Master 127.0.97.60:40409 requested a full tablet report, sending...
I20260926 03:00:13.171291 7134 ts_manager.cc:194] Registered new tserver with Master: 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:13.171954 7324 ts_manager.cc:194] Registered new tserver with Master: 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:13.182102 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.130378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.0.97.62:42493)
0926 03:00:13.145402 (+ 15024us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.145407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.160548 (+ 15141us) server_negotiation.cc:407] Connection header received
0926 03:00:13.160769 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.160774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.160875 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.160975 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.164270 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.165368 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.168062 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.168707 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.168842 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.174077 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.174085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.174091 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.174134 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.177642 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.178561 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.178568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.178572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.179168 (+ 596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.180464 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.180471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.180477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.180843 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.181130 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.181472 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.181726 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14931,"thread_start_us":30377,"threads_started":1}
I20260926 03:00:13.182911 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.130423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.97.61:34195)
0926 03:00:13.151439 (+ 21016us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.151445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.163673 (+ 12228us) server_negotiation.cc:407] Connection header received
0926 03:00:13.163895 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.163900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.163965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.164034 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.167616 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.170300 (+ 2684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.171871 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.172572 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.172699 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.175759 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.175764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.175768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.175799 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.179138 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.179840 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.179845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.179849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.180041 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.180387 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.180394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.180398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.180766 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.181043 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.182296 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.182513 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20865,"thread_start_us":18701,"threads_started":1}
I20260926 03:00:13.186390 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.152115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.0.97.60:40409)
0926 03:00:13.163230 (+ 11115us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.163236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.167656 (+ 4420us) server_negotiation.cc:407] Connection header received
0926 03:00:13.167832 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.167839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.167924 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.168017 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.172706 (+ 4689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.173707 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.175705 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.176420 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.176526 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.180045 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.180049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.180052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.180085 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.183265 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.184021 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.184026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.184030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.184204 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.184647 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.184652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.184655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.185072 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.185311 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.185942 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.186158 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11030,"thread_start_us":15838,"threads_started":1}
I20260926 03:00:13.186722 6537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:00:13.193780 6537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:13.234902 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.212016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.0.97.2:44699)
0926 03:00:13.224547 (+ 12531us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.224561 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.227781 (+ 3220us) server_negotiation.cc:407] Connection header received
0926 03:00:13.227879 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.227885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.228116 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.228312 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:13.228856 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.230346 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.232409 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.233888 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.234043 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.234262 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.234472 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.234624 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12301,"thread_start_us":12178,"threads_started":1}
I20260926 03:00:13.234982 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.211773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44699 (local address 127.0.0.1:40348)
0926 03:00:13.226867 (+ 15094us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.226889 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.227022 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.228317 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.228324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.228355 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:13.228678 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.228691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.230508 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.230513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.232270 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.232281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.234096 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.234106 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.234295 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.234335 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.234427 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.234565 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14946,"mutex_wait_us":24,"thread_start_us":14440,"threads_started":1}
I20260926 03:00:13.236213 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.211580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.0.97.1:39395)
0926 03:00:13.226039 (+ 14459us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.226051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.226586 (+ 535us) server_negotiation.cc:407] Connection header received
0926 03:00:13.226838 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.226848 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.227134 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.227378 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:13.228049 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.229399 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.233433 (+ 4034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.234964 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.235099 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.235275 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.235782 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.235929 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14241,"thread_start_us":14464,"threads_started":1}
I20260926 03:00:13.236250 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.211387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39395 (local address 127.0.0.1:53400)
0926 03:00:13.226459 (+ 15072us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.226477 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.226641 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.227372 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.227379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.227418 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:13.227875 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.227889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.229563 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.229568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.233267 (+ 3699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.233281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.235402 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.235407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.235586 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.235622 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.235745 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.235886 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14963,"mutex_wait_us":1,"thread_start_us":15373,"threads_started":1}
I20260926 03:00:13.239511 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.212983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44007 (local address 127.0.0.1:34932)
0926 03:00:13.230255 (+ 17272us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.230287 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.230440 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.231014 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.231017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.231047 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:13.231437 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.231469 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.233883 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.233890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.235880 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.235892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.238801 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.238807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.238977 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.239006 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.239103 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.239216 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17145,"thread_start_us":16968,"threads_started":1}
I20260926 03:00:13.239723 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.213730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.0.97.3:44007)
0926 03:00:13.229757 (+ 16027us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.229768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.230440 (+ 672us) server_negotiation.cc:407] Connection header received
0926 03:00:13.230569 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.230576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.230821 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.231033 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:13.232072 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.233763 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.237101 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.238592 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.238723 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.238902 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.239150 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.239289 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15827,"thread_start_us":15653,"threads_started":1}
I20260926 03:00:13.239329 7583 tablet_service.cc:1467] Processing CreateTablet for tablet f3681c881a914a2caa6361c4c48f2422 (DEFAULT_TABLE table=test-table [id=c9e6212940aa45f9a9b53985ea4be877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:13.239827 7438 tablet_service.cc:1467] Processing CreateTablet for tablet f3681c881a914a2caa6361c4c48f2422 (DEFAULT_TABLE table=test-table [id=c9e6212940aa45f9a9b53985ea4be877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:13.240679 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3681c881a914a2caa6361c4c48f2422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:13.241058 7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3681c881a914a2caa6361c4c48f2422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:13.250720 7725 tablet_service.cc:1467] Processing CreateTablet for tablet f3681c881a914a2caa6361c4c48f2422 (DEFAULT_TABLE table=test-table [id=c9e6212940aa45f9a9b53985ea4be877]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:13.252024 7824 tablet_bootstrap.cc:492] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Bootstrap starting.
I20260926 03:00:13.252072 7725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3681c881a914a2caa6361c4c48f2422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:13.256016 7824 tablet_bootstrap.cc:654] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:13.257236 7824 log.cc:824] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:13.258574 7825 tablet_bootstrap.cc:492] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Bootstrap starting.
I20260926 03:00:13.260499 7824 tablet_bootstrap.cc:492] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: No bootstrap required, opened a new log
I20260926 03:00:13.260753 7824 ts_tablet_manager.cc:1397] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:00:13.261965 7825 tablet_bootstrap.cc:654] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:13.263489 7825 log.cc:824] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:13.266461 7825 tablet_bootstrap.cc:492] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: No bootstrap required, opened a new log
I20260926 03:00:13.266768 7825 ts_tablet_manager.cc:1397] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260926 03:00:13.267225 7828 tablet_bootstrap.cc:492] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Bootstrap starting.
I20260926 03:00:13.270506 7828 tablet_bootstrap.cc:654] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:13.271497 7828 log.cc:824] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:13.272733 7824 raft_consensus.cc:348] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.273204 7824 raft_consensus.cc:374] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:13.273301 7824 raft_consensus.cc:729] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd226c00ab9141e998c7d13428d6b268, State: Initialized, Role: FOLLOWER
I20260926 03:00:13.273908 7824 consensus_queue.cc:260] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.274125 7828 tablet_bootstrap.cc:492] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: No bootstrap required, opened a new log
I20260926 03:00:13.274405 7828 ts_tablet_manager.cc:1397] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 03:00:13.276046 7824 ts_tablet_manager.cc:1428] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 03:00:13.277743 7825 raft_consensus.cc:348] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.278206 7825 raft_consensus.cc:374] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:13.278353 7825 raft_consensus.cc:729] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd03a6db3c494566b954e4a9bc73d704, State: Initialized, Role: FOLLOWER
I20260926 03:00:13.279062 7825 consensus_queue.cc:260] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.281246 7825 ts_tablet_manager.cc:1428] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 03:00:13.281571 7649 heartbeater.cc:503] Master 127.0.97.62:42493 was elected leader, sending a full tablet report...
I20260926 03:00:13.288758 7828 raft_consensus.cc:348] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.289146 7828 raft_consensus.cc:374] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:13.289218 7828 raft_consensus.cc:729] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb395bdd48542e1baa56277b11090f2, State: Initialized, Role: FOLLOWER
I20260926 03:00:13.289729 7828 consensus_queue.cc:260] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.290688 7791 heartbeater.cc:503] Master 127.0.97.62:42493 was elected leader, sending a full tablet report...
I20260926 03:00:13.291280 7828 ts_tablet_manager.cc:1428] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Time spent starting tablet: real 0.017s user 0.010s sys 0.000s
W20260926 03:00:13.378206 7793 tablet.cc:2369] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:00:13.454483 7506 tablet.cc:2369] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:00:13.529330 7651 tablet.cc:2369] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:00:13.611881 7830 raft_consensus.cc:482] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:13.612164 7830 raft_consensus.cc:504] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.614395 7830 leader_election.cc:288] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699), 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:13.621771 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.614563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44699 (local address 127.0.97.1:49301)
0926 03:00:13.615543 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.615571 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.615739 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.616203 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.616208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.616238 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:13.616487 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.616496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.617746 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.617752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.619524 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.619538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.620977 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.620984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.621174 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.621219 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.621303 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.621432 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":19,"thread_start_us":530,"threads_started":1}
I20260926 03:00:13.621793 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.614801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49301 (local address 127.0.97.2:44699)
0926 03:00:13.615187 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.615193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.615693 (+ 500us) server_negotiation.cc:407] Connection header received
0926 03:00:13.615908 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.615915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.616014 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.616119 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:13.616635 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.617612 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.619701 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.620771 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.620873 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.620945 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.621363 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.621511 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260926 03:00:13.622123 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.614948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35087 (local address 127.0.97.3:44007)
0926 03:00:13.615345 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.615353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.616044 (+ 691us) server_negotiation.cc:407] Connection header received
0926 03:00:13.616271 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.616279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.616360 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.616413 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:13.617162 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.617925 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.620257 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.621122 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.621242 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.621351 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.621730 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.621868 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260926 03:00:13.622205 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.614807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44007 (local address 127.0.97.1:35087)
0926 03:00:13.615922 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:13.615952 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:13.616086 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:13.616595 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:13.616599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:13.616626 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:13.616965 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.616977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.618068 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.618074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.620097 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:13.620115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.621362 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.621370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:13.621548 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.621584 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:00:13.621687 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:00:13.621840 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":695,"threads_started":1}
I20260926 03:00:13.623051 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3681c881a914a2caa6361c4c48f2422" candidate_uuid: "fd226c00ab9141e998c7d13428d6b268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd03a6db3c494566b954e4a9bc73d704" is_pre_election: true
I20260926 03:00:13.623162 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3681c881a914a2caa6361c4c48f2422" candidate_uuid: "fd226c00ab9141e998c7d13428d6b268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb395bdd48542e1baa56277b11090f2" is_pre_election: true
I20260926 03:00:13.623955 7603 raft_consensus.cc:2454] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd226c00ab9141e998c7d13428d6b268 in term 0.
I20260926 03:00:13.624033 7745 raft_consensus.cc:2454] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd226c00ab9141e998c7d13428d6b268 in term 0.
I20260926 03:00:13.624672 7394 leader_election.cc:302] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb395bdd48542e1baa56277b11090f2, bd03a6db3c494566b954e4a9bc73d704, fd226c00ab9141e998c7d13428d6b268; no voters:
I20260926 03:00:13.625113 7830 raft_consensus.cc:2781] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:13.625267 7830 raft_consensus.cc:482] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:13.625372 7830 raft_consensus.cc:3037] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:13.627650 7830 raft_consensus.cc:504] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.628320 7830 leader_election.cc:288] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [CANDIDATE]: Term 1 election: Requested vote from peers bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699), 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:13.628664 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3681c881a914a2caa6361c4c48f2422" candidate_uuid: "fd226c00ab9141e998c7d13428d6b268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd03a6db3c494566b954e4a9bc73d704"
I20260926 03:00:13.628793 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3681c881a914a2caa6361c4c48f2422" candidate_uuid: "fd226c00ab9141e998c7d13428d6b268" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb395bdd48542e1baa56277b11090f2"
I20260926 03:00:13.628875 7603 raft_consensus.cc:3037] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:13.629027 7745 raft_consensus.cc:3037] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:13.632349 7603 raft_consensus.cc:2454] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd226c00ab9141e998c7d13428d6b268 in term 1.
I20260926 03:00:13.632396 7745 raft_consensus.cc:2454] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd226c00ab9141e998c7d13428d6b268 in term 1.
I20260926 03:00:13.632774 7391 leader_election.cc:302] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd03a6db3c494566b954e4a9bc73d704, fd226c00ab9141e998c7d13428d6b268; no voters:
I20260926 03:00:13.633131 7830 raft_consensus.cc:2781] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:13.633625 7830 raft_consensus.cc:686] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 1 LEADER]: Becoming Leader. State: Replica: fd226c00ab9141e998c7d13428d6b268, State: Running, Role: LEADER
I20260926 03:00:13.634207 7830 consensus_queue.cc:237] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:13.639206 6536 catalog_manager.cc:5495] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd226c00ab9141e998c7d13428d6b268 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fd226c00ab9141e998c7d13428d6b268" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } health_report { overall_health: UNKNOWN } } }
I20260926 03:00:13.682430 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.676931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.0.97.1:39395)
0926 03:00:13.677179 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.677186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.677290 (+ 104us) server_negotiation.cc:407] Connection header received
0926 03:00:13.677569 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.677575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.677677 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.677835 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:13.678215 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.678934 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.680242 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.680633 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.680785 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.681301 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:13.681885 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:13.682043 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.682096 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.682172 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 03:00:13.690462 7603 raft_consensus.cc:1261] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 1 FOLLOWER]: Refusing update from remote peer fd226c00ab9141e998c7d13428d6b268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:13.690462 7745 raft_consensus.cc:1261] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Refusing update from remote peer fd226c00ab9141e998c7d13428d6b268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:13.691469 7835 consensus_queue.cc:1035] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:13.691795 7830 consensus_queue.cc:1035] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:13.705965 7838 mvcc.cc:205] Tried to move back new op lower bound from 7333444049666072576 to 7333444049449472000. Current Snapshot: MvccSnapshot[applied={T|T < 7333444049666072576}]
I20260926 03:00:13.720920 6537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:13.721225 6537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:13.727807 6537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a80c50d5c0c445a6bf3709ee0eecbb26: Sending DeleteTablet for 3 replicas of tablet f3681c881a914a2caa6361c4c48f2422
I20260926 03:00:13.728806 7438 tablet_service.cc:1514] Processing DeleteTablet for tablet f3681c881a914a2caa6361c4c48f2422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53400
I20260926 03:00:13.728965 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet f3681c881a914a2caa6361c4c48f2422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40348
I20260926 03:00:13.729806 7725 tablet_service.cc:1514] Processing DeleteTablet for tablet f3681c881a914a2caa6361c4c48f2422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:13 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34932
I20260926 03:00:13.741912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6505
I20260926 03:00:13.743758 7848 tablet_replica.cc:330] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: stopping tablet replica
I20260926 03:00:13.743834 6567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:13.744284 7848 raft_consensus.cc:2229] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:00:13.745085 7848 raft_consensus.cc:2258] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:13.749688 7848 ts_tablet_manager.cc:1905] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:13.750680 7850 tablet_replica.cc:330] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: stopping tablet replica
I20260926 03:00:13.751313 7850 raft_consensus.cc:2229] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:13.752925 7850 raft_consensus.cc:2258] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:13.753415 7849 tablet_replica.cc:330] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: stopping tablet replica
I20260926 03:00:13.753834 7849 raft_consensus.cc:2229] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:13.754276 7849 raft_consensus.cc:2258] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:13.755389 7850 ts_tablet_manager.cc:1905] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:13.756021 7849 ts_tablet_manager.cc:1905] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:13.762784 7850 ts_tablet_manager.cc:1918] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:13.762957 7850 log.cc:1196] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/f3681c881a914a2caa6361c4c48f2422
I20260926 03:00:13.763244 7849 ts_tablet_manager.cc:1918] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:13.763393 7849 log.cc:1196] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/f3681c881a914a2caa6361c4c48f2422
I20260926 03:00:13.763893 7850 ts_tablet_manager.cc:1939] T f3681c881a914a2caa6361c4c48f2422 P 5cb395bdd48542e1baa56277b11090f2: Deleting consensus metadata
I20260926 03:00:13.764277 7849 ts_tablet_manager.cc:1939] T f3681c881a914a2caa6361c4c48f2422 P bd03a6db3c494566b954e4a9bc73d704: Deleting consensus metadata
I20260926 03:00:13.766968 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.733141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.0.97.61:34195)
0926 03:00:13.749487 (+ 16346us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.749496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.749521 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:00:13.749595 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.749603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.749700 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.749978 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.751766 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.753181 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.755298 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.756004 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.759426 (+ 3422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.759631 (+ 205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.759636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.759641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.759690 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.763551 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.764649 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.764654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.764658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.764822 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.765283 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.765287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.765291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.765655 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.765927 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.766423 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.766630 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16162,"thread_start_us":16674,"threads_started":1}
I20260926 03:00:13.774751 7848 ts_tablet_manager.cc:1918] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:13.775015 7848 log.cc:1196] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/f3681c881a914a2caa6361c4c48f2422
I20260926 03:00:13.775712 7848 ts_tablet_manager.cc:1939] T f3681c881a914a2caa6361c4c48f2422 P fd226c00ab9141e998c7d13428d6b268: Deleting consensus metadata
I20260926 03:00:13.782917 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.739227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.0.97.60:40409)
0926 03:00:13.739746 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.739753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.767026 (+ 27273us) server_negotiation.cc:407] Connection header received
0926 03:00:13.767482 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.767488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.767575 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.767692 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.769331 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.770562 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.771804 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.772230 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.772374 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.775591 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.775598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.775602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.775642 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.780371 (+ 4729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.781030 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.781035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.781039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.781163 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.781555 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.781560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.781564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.781829 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.782090 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:13.782471 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 03:00:13.782649 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260926 03:00:13.948263 6524 catalog_manager.cc:4841] TS 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007): tablet f3681c881a914a2caa6361c4c48f2422 (table test-table [id=c9e6212940aa45f9a9b53985ea4be877]) successfully deleted
I20260926 03:00:13.948824 6522 catalog_manager.cc:4841] TS fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395): tablet f3681c881a914a2caa6361c4c48f2422 (table test-table [id=c9e6212940aa45f9a9b53985ea4be877]) successfully deleted
I20260926 03:00:13.948849 6521 catalog_manager.cc:4841] TS bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699): tablet f3681c881a914a2caa6361c4c48f2422 (table test-table [id=c9e6212940aa45f9a9b53985ea4be877]) successfully deleted
W20260926 03:00:13.949404 6521 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:44699, user_credentials={real_user=slave}} blocked reactor thread for 183306us
W20260926 03:00:13.948563 6524 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:44007, user_credentials={real_user=slave}} blocked reactor thread for 182822us
I20260926 03:00:13.949990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6505
I20260926 03:00:14.000739 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.983049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.97.61:34195)
0926 03:00:13.983254 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.983261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.983832 (+ 571us) server_negotiation.cc:407] Connection header received
0926 03:00:13.984089 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.984096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.984175 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.984241 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.986735 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.987828 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.990146 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.990820 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.990920 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.994605 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.994611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.994616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.994656 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:13.997890 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.998708 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.998715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.998718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.998856 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:13.999293 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:13.999307 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:13.999310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:13.999618 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:13.999903 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:14.000211 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 03:00:14.000416 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:00:14.005388 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:13.984021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.97.60:40409)
0926 03:00:13.984247 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:13.984255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:13.991069 (+ 6814us) server_negotiation.cc:407] Connection header received
0926 03:00:13.991760 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:13.991766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:13.991860 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:13.991932 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:13.993653 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.994307 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.996052 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:13.996744 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:13.996871 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:13.999538 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:13.999545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:13.999549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:13.999585 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:14.002944 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:14.003515 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:14.003520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:14.003524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:14.003694 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:14.004006 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:14.004010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:14.004014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:14.004340 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:14.004554 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:14.004801 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 03:00:14.005054 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:00:14.710848 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:14.710152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.1:43219)
0926 03:00:14.710591 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:14.710696 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
W20260926 03:00:14.711225 7504 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:14.713932 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:14.713135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.3:54827)
0926 03:00:14.713554 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:14.713703 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":194,"threads_started":1}
W20260926 03:00:14.714358 7791 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:14.719506 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:14.718671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:43787)
0926 03:00:14.719142 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:14.719292 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
W20260926 03:00:14.719955 7649 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:15.227138 7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80c50d5c0c445a6bf3709ee0eecbb26)
I20260926 03:00:15.227458 7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } }
I20260926 03:00:15.228253 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a80c50d5c0c445a6bf3709ee0eecbb26)
I20260926 03:00:15.228466 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } }
I20260926 03:00:15.229028 7865 leader_election.cc:288] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80c50d5c0c445a6bf3709ee0eecbb26 (127.0.97.62:42493), ae770f4b38954985b59301605d8d61f5 (127.0.97.61:34195)
I20260926 03:00:15.229924 7866 leader_election.cc:288] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a80c50d5c0c445a6bf3709ee0eecbb26 (127.0.97.62:42493), bb9850d450f64613a023c5b3e841da85 (127.0.97.60:40409)
I20260926 03:00:15.230844 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.229107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46564)
0926 03:00:15.230461 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.230618 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1248,"thread_start_us":1117,"threads_started":1}
W20260926 03:00:15.231170 7308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:15.248252 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.241373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46574)
0926 03:00:15.247818 (+ 6445us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.248003 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6339,"threads_started":1}
W20260926 03:00:15.248709 7118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:15.253299 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.229258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34195 (local address 127.0.0.1:52176)
0926 03:00:15.231038 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.231056 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:15.231214 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:15.242525 (+ 11311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:15.242533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:15.242581 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:15.242918 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.242933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.247026 (+ 4093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.247034 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.249760 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.249778 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.252318 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.252323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.252574 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.252616 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:00:15.252741 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 03:00:15.252891 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"mutex_wait_us":1,"thread_start_us":11398,"threads_started":1}
I20260926 03:00:15.253434 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.229409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.0.97.61:34195)
0926 03:00:15.241651 (+ 12242us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.241660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.241685 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:00:15.241777 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.241783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.241881 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.243104 (+ 1223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:15.243350 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.244880 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.250194 (+ 5314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.252122 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.252283 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.252369 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.252899 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.253070 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11724,"thread_start_us":11583,"threads_started":1}
I20260926 03:00:15.253778 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.253308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46576)
0926 03:00:15.253467 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.253587 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:00:15.254034 7308 leader_election.cc:334] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a80c50d5c0c445a6bf3709ee0eecbb26 (127.0.97.62:42493): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:15.254498 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.254058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46592)
0926 03:00:15.254289 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.254372 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 03:00:15.254576 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb9850d450f64613a023c5b3e841da85" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae770f4b38954985b59301605d8d61f5" is_pre_election: true
W20260926 03:00:15.254741 7118 leader_election.cc:334] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a80c50d5c0c445a6bf3709ee0eecbb26 (127.0.97.62:42493): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:15.254930 7144 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9850d450f64613a023c5b3e841da85 in term 1.
I20260926 03:00:15.255429 7310 leader_election.cc:302] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae770f4b38954985b59301605d8d61f5, bb9850d450f64613a023c5b3e841da85; no voters: a80c50d5c0c445a6bf3709ee0eecbb26
I20260926 03:00:15.255810 7865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 03:00:15.255949 7865 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a80c50d5c0c445a6bf3709ee0eecbb26)
I20260926 03:00:15.256062 7865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:00:15.257019 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.231801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40409 (local address 127.0.0.1:59808)
0926 03:00:15.247249 (+ 15448us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.247271 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:15.247416 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:15.248513 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:15.248517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:15.248540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:15.248943 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.248956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.251040 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.251049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.254475 (+ 3426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.254488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.256275 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.256279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.256395 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.256417 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:00:15.256506 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:00:15.256626 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15277,"mutex_wait_us":860,"thread_start_us":18083,"threads_started":1}
I20260926 03:00:15.257062 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.231786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.0.97.60:40409)
0926 03:00:15.247906 (+ 16120us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.247912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.247935 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:00:15.247991 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.247998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.248071 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.248161 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:15.249153 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.250837 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.254632 (+ 3795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.255845 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.256041 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.256178 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.256550 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.256678 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15997,"thread_start_us":16602,"threads_started":1}
I20260926 03:00:15.257629 7334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae770f4b38954985b59301605d8d61f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bb9850d450f64613a023c5b3e841da85" is_pre_election: true
I20260926 03:00:15.258728 7865 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } }
I20260926 03:00:15.259114 7334 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ae770f4b38954985b59301605d8d61f5 in current term 2: Already voted for candidate bb9850d450f64613a023c5b3e841da85 in this term.
I20260926 03:00:15.259363 7865 leader_election.cc:288] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [CANDIDATE]: Term 2 election: Requested vote from peers a80c50d5c0c445a6bf3709ee0eecbb26 (127.0.97.62:42493), ae770f4b38954985b59301605d8d61f5 (127.0.97.61:34195)
I20260926 03:00:15.259754 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.259391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46600)
0926 03:00:15.259531 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.259605 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 03:00:15.259852 7144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb9850d450f64613a023c5b3e841da85" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae770f4b38954985b59301605d8d61f5"
I20260926 03:00:15.259989 7120 leader_election.cc:302] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae770f4b38954985b59301605d8d61f5; no voters: a80c50d5c0c445a6bf3709ee0eecbb26, bb9850d450f64613a023c5b3e841da85
I20260926 03:00:15.260128 7144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:00:15.261308 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.260902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46604)
0926 03:00:15.261087 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.261169 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 03:00:15.261513 7308 leader_election.cc:334] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a80c50d5c0c445a6bf3709ee0eecbb26 (127.0.97.62:42493): Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
I20260926 03:00:15.262516 7144 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9850d450f64613a023c5b3e841da85 in term 2.
I20260926 03:00:15.262669 7866 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:00:15.262889 7310 leader_election.cc:302] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae770f4b38954985b59301605d8d61f5, bb9850d450f64613a023c5b3e841da85; no voters: a80c50d5c0c445a6bf3709ee0eecbb26
I20260926 03:00:15.263183 7865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:00:15.263579 7865 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [term 2 LEADER]: Becoming Leader. State: Replica: bb9850d450f64613a023c5b3e841da85, State: Running, Role: LEADER
I20260926 03:00:15.263927 7865 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } }
I20260926 03:00:15.265600 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9850d450f64613a023c5b3e841da85. Latest consensus state: current_term: 2 leader_uuid: "bb9850d450f64613a023c5b3e841da85" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:15.265892 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:15.266494 7877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:15.269797 7877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:15.270617 7877 catalog_manager.cc:1259] Loaded cluster ID: 610be077267b48729512193805fe9b5c
I20260926 03:00:15.270696 7877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:15.271739 7877 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:15.272468 7877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Loaded TSK: 0
I20260926 03:00:15.273295 7877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:00:15.281910 7324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:00:15.288721 7324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:15.292618 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.292241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46622)
0926 03:00:15.292405 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.292499 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:00:15.292706 7144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [term 2 FOLLOWER]: Refusing update from remote peer bb9850d450f64613a023c5b3e841da85: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 03:00:15.293562 7875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 03:00:15.293731 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.293339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46634)
0926 03:00:15.293499 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.293590 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:00:15.293895 7308 consensus_peers.cc:487] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 -> Peer a80c50d5c0c445a6bf3709ee0eecbb26 (127.0.97.62:42493): Couldn't send request to peer a80c50d5c0c445a6bf3709ee0eecbb26. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:00:15.295853 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb9850d450f64613a023c5b3e841da85. Latest consensus state: current_term: 2 leader_uuid: "bb9850d450f64613a023c5b3e841da85" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:15.296137 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:00:15.297602 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bb9850d450f64613a023c5b3e841da85" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:15.297837 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:15.297986 7879 mvcc.cc:205] Tried to move back new op lower bound from 7333444056232566784 to 7333444056124182528. Current Snapshot: MvccSnapshot[applied={T|T < 7333444056232566784}]
I20260926 03:00:15.298576 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bb9850d450f64613a023c5b3e841da85" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a80c50d5c0c445a6bf3709ee0eecbb26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42493 } } peers { permanent_uuid: "ae770f4b38954985b59301605d8d61f5" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "bb9850d450f64613a023c5b3e841da85" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40409 } attrs { promote: false } } }
I20260926 03:00:15.298815 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae770f4b38954985b59301605d8d61f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:00:15.313059 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.306114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44699 (local address 127.0.0.1:50260)
0926 03:00:15.306233 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.306248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:15.306326 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:15.307689 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:15.307694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:15.307717 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:15.307869 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.307878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.309385 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.309396 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.310981 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.310993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.312305 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.312310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.312494 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.312517 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:00:15.312612 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:00:15.312745 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 03:00:15.313077 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.306252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.0.97.2:44699)
0926 03:00:15.307389 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.307395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.307413 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:00:15.307476 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.307481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.307544 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.307989 (+ 445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:15.308122 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.309224 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.311163 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.312156 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.312294 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.312390 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.312645 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.312771 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":909,"threads_started":1}
I20260926 03:00:15.313452 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.305324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39395 (local address 127.0.0.1:51490)
0926 03:00:15.305451 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.305469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:15.305587 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:15.306519 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:15.306524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:15.306546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:15.306963 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.306977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.308253 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.308259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.310318 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.310331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.312461 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.312470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.312739 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.312792 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:00:15.312879 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:00:15.313080 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 03:00:15.314015 7583 tablet_service.cc:1467] Processing CreateTablet for tablet 3574bf49e149450789101db0c01cc3a2 (DEFAULT_TABLE table=test-table [id=a37d5899611f424fb5b807c499a3f189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:15.314436 7583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3574bf49e149450789101db0c01cc3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:15.314615 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.305434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.0.97.1:39395)
0926 03:00:15.305811 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.305816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.305830 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:00:15.305915 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.305923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.306023 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.306473 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:15.307090 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.308107 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.310489 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.312170 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.313833 (+ 1663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.313993 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.314052 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.314236 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":190,"threads_started":1}
I20260926 03:00:15.318065 7438 tablet_service.cc:1467] Processing CreateTablet for tablet 3574bf49e149450789101db0c01cc3a2 (DEFAULT_TABLE table=test-table [id=a37d5899611f424fb5b807c499a3f189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:15.318429 7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3574bf49e149450789101db0c01cc3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:15.322824 7888 tablet_bootstrap.cc:492] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: Bootstrap starting.
I20260926 03:00:15.323289 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.306435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44007 (local address 127.0.0.1:50956)
0926 03:00:15.309185 (+ 2750us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.309206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:15.309340 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:15.318097 (+ 8757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:15.318102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:15.318126 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:15.318422 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.318430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.319413 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.319421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.321121 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.321134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.322590 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.322598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.322764 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.322808 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:00:15.322868 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 03:00:15.322984 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":574,"threads_started":1}
I20260926 03:00:15.324255 7888 tablet_bootstrap.cc:654] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:15.324290 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.306927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.0.97.3:44007)
0926 03:00:15.317854 (+ 10927us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.317859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.317875 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:00:15.317928 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.317931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.317981 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.318087 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:15.318566 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.319289 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.321274 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.322388 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.323707 (+ 1319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.323802 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.323852 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.323996 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10801,"thread_start_us":11269,"threads_started":1}
I20260926 03:00:15.324848 7889 tablet_bootstrap.cc:492] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: Bootstrap starting.
I20260926 03:00:15.324965 7725 tablet_service.cc:1467] Processing CreateTablet for tablet 3574bf49e149450789101db0c01cc3a2 (DEFAULT_TABLE table=test-table [id=a37d5899611f424fb5b807c499a3f189]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:15.325431 7725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3574bf49e149450789101db0c01cc3a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:15.326546 7888 tablet_bootstrap.cc:492] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: No bootstrap required, opened a new log
I20260926 03:00:15.326731 7888 ts_tablet_manager.cc:1397] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 03:00:15.326787 7889 tablet_bootstrap.cc:654] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:15.327556 7888 raft_consensus.cc:348] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.327842 7888 raft_consensus.cc:374] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:15.327947 7888 raft_consensus.cc:729] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd03a6db3c494566b954e4a9bc73d704, State: Initialized, Role: FOLLOWER
I20260926 03:00:15.328210 7888 consensus_queue.cc:260] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.329128 7889 tablet_bootstrap.cc:492] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: No bootstrap required, opened a new log
I20260926 03:00:15.329295 7889 ts_tablet_manager.cc:1397] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 03:00:15.330263 7889 raft_consensus.cc:348] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.330566 7889 raft_consensus.cc:374] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:15.330680 7889 raft_consensus.cc:729] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd226c00ab9141e998c7d13428d6b268, State: Initialized, Role: FOLLOWER
I20260926 03:00:15.330964 7889 consensus_queue.cc:260] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.331159 7893 tablet_bootstrap.cc:492] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: Bootstrap starting.
I20260926 03:00:15.331993 7889 ts_tablet_manager.cc:1428] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:00:15.333524 7893 tablet_bootstrap.cc:654] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:15.335085 7888 ts_tablet_manager.cc:1428] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20260926 03:00:15.335532 7893 tablet_bootstrap.cc:492] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: No bootstrap required, opened a new log
I20260926 03:00:15.335652 7893 ts_tablet_manager.cc:1397] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 03:00:15.336154 7503 heartbeater.cc:503] Master 127.0.97.60:40409 was elected leader, sending a full tablet report...
I20260926 03:00:15.336639 7893 raft_consensus.cc:348] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.336930 7893 raft_consensus.cc:374] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:15.337041 7893 raft_consensus.cc:729] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb395bdd48542e1baa56277b11090f2, State: Initialized, Role: FOLLOWER
I20260926 03:00:15.337316 7893 consensus_queue.cc:260] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.338331 7648 heartbeater.cc:503] Master 127.0.97.60:40409 was elected leader, sending a full tablet report...
I20260926 03:00:15.341967 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.335594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:37131)
0926 03:00:15.341279 (+ 5685us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.341472 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5507,"thread_start_us":5341,"threads_started":1}
I20260926 03:00:15.343227 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.342733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:54497)
0926 03:00:15.342951 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.343055 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 03:00:15.344385 7893 ts_tablet_manager.cc:1428] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: Time spent starting tablet: real 0.009s user 0.003s sys 0.006s
I20260926 03:00:15.345518 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.344395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.3:59407)
0926 03:00:15.345000 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.345219 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":251,"threads_started":1}
I20260926 03:00:15.346105 7790 heartbeater.cc:503] Master 127.0.97.60:40409 was elected leader, sending a full tablet report...
I20260926 03:00:15.348325 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.347719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.3:53701)
0926 03:00:15.348057 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.348161 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260926 03:00:15.349550 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.332616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.1:37355)
0926 03:00:15.349255 (+ 16639us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.349342 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16548,"threads_started":1}
I20260926 03:00:15.350438 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.350167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.1:35741)
0926 03:00:15.350286 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.350347 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 03:00:15.442031 7892 raft_consensus.cc:482] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:15.442281 7892 raft_consensus.cc:504] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.443746 7892 leader_election.cc:288] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395), 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:15.449395 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.444077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60621 (local address 127.0.97.1:39395)
0926 03:00:15.444265 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.444271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.444406 (+ 135us) server_negotiation.cc:407] Connection header received
0926 03:00:15.444646 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.444653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.444723 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.444819 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:15.445354 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.446021 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.447684 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.448415 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.448577 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.448681 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.449025 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.449142 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:00:15.449430 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.443983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39395 (local address 127.0.97.2:60621)
0926 03:00:15.444285 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.444304 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:15.444510 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:15.444846 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:15.444850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:15.444882 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:15.445217 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.445225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.446157 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.446165 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.447536 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.447547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.448611 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.448616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.448806 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.448851 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 03:00:15.448977 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 03:00:15.449129 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":3}
I20260926 03:00:15.449591 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.444165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42061 (local address 127.0.97.3:44007)
0926 03:00:15.444380 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.444388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.444722 (+ 334us) server_negotiation.cc:407] Connection header received
0926 03:00:15.444935 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.444939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.444991 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.445121 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:15.445597 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.446272 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.447833 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.448546 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.448727 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.448842 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.449222 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.449337 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:00:15.449919 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.444019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44007 (local address 127.0.97.2:42061)
0926 03:00:15.444625 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.444638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:15.444782 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:15.445135 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:15.445139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:15.445164 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:15.445459 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.445467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.446402 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.446407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.447687 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:15.447697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.448811 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.448818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:15.449079 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.449102 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:00:15.449199 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:00:15.449322 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":41,"thread_start_us":378,"threads_started":1}
I20260926 03:00:15.450198 7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3574bf49e149450789101db0c01cc3a2" candidate_uuid: "bd03a6db3c494566b954e4a9bc73d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd226c00ab9141e998c7d13428d6b268" is_pre_election: true
I20260926 03:00:15.450557 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3574bf49e149450789101db0c01cc3a2" candidate_uuid: "bd03a6db3c494566b954e4a9bc73d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb395bdd48542e1baa56277b11090f2" is_pre_election: true
I20260926 03:00:15.450609 7458 raft_consensus.cc:2454] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd03a6db3c494566b954e4a9bc73d704 in term 0.
I20260926 03:00:15.450762 7745 raft_consensus.cc:2454] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd03a6db3c494566b954e4a9bc73d704 in term 0.
I20260926 03:00:15.451146 7537 leader_election.cc:302] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd03a6db3c494566b954e4a9bc73d704, fd226c00ab9141e998c7d13428d6b268; no voters:
I20260926 03:00:15.451457 7892 raft_consensus.cc:2781] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:15.451574 7892 raft_consensus.cc:482] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:15.451682 7892 raft_consensus.cc:3037] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:15.453965 7892 raft_consensus.cc:504] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.454609 7892 leader_election.cc:288] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [CANDIDATE]: Term 1 election: Requested vote from peers fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395), 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007)
I20260926 03:00:15.454939 7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3574bf49e149450789101db0c01cc3a2" candidate_uuid: "bd03a6db3c494566b954e4a9bc73d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd226c00ab9141e998c7d13428d6b268"
I20260926 03:00:15.455027 7745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3574bf49e149450789101db0c01cc3a2" candidate_uuid: "bd03a6db3c494566b954e4a9bc73d704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb395bdd48542e1baa56277b11090f2"
I20260926 03:00:15.455134 7458 raft_consensus.cc:3037] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:15.455199 7745 raft_consensus.cc:3037] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:15.458102 7745 raft_consensus.cc:2454] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd03a6db3c494566b954e4a9bc73d704 in term 1.
I20260926 03:00:15.458115 7458 raft_consensus.cc:2454] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd03a6db3c494566b954e4a9bc73d704 in term 1.
I20260926 03:00:15.458452 7539 leader_election.cc:302] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb395bdd48542e1baa56277b11090f2, bd03a6db3c494566b954e4a9bc73d704; no voters:
I20260926 03:00:15.458740 7892 raft_consensus.cc:2781] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:15.459187 7892 raft_consensus.cc:686] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 1 LEADER]: Becoming Leader. State: Replica: bd03a6db3c494566b954e4a9bc73d704, State: Running, Role: LEADER
I20260926 03:00:15.459579 7892 consensus_queue.cc:237] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } }
I20260926 03:00:15.460101 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.459700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:57151)
0926 03:00:15.459838 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.459959 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:00:15.463016 7323 catalog_manager.cc:5495] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bd03a6db3c494566b954e4a9bc73d704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd03a6db3c494566b954e4a9bc73d704" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 } health_report { overall_health: UNKNOWN } } }
I20260926 03:00:15.505625 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.500228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.0.97.2:44699)
0926 03:00:15.500420 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.500424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.500441 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:00:15.500610 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.500614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.500686 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.500808 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:15.501203 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.501842 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.503599 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.503974 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.504076 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.504531 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:15.505080 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:15.505189 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.505299 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.505357 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:00:15.509477 7745 raft_consensus.cc:1261] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Refusing update from remote peer bd03a6db3c494566b954e4a9bc73d704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:15.509873 7458 raft_consensus.cc:1261] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 1 FOLLOWER]: Refusing update from remote peer bd03a6db3c494566b954e4a9bc73d704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:15.510186 7901 consensus_queue.cc:1035] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb395bdd48542e1baa56277b11090f2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:15.510620 7892 consensus_queue.cc:1035] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd226c00ab9141e998c7d13428d6b268" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:15.513019 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.512526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.3:58827)
0926 03:00:15.512731 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.512825 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 03:00:15.513775 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.513313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.1:56851)
0926 03:00:15.513493 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.513578 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:00:15.518771 7904 mvcc.cc:205] Tried to move back new op lower bound from 7333444057120587776 to 7333444056926326784. Current Snapshot: MvccSnapshot[applied={T|T < 7333444057120587776}]
I20260926 03:00:15.518904 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.518485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.3:46981)
0926 03:00:15.518675 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.518762 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:00:15.519959 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.519347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:39129)
0926 03:00:15.519678 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.519776 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260926 03:00:15.531504 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.530743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.2:49107)
0926 03:00:15.531224 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.531338 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
I20260926 03:00:15.534004 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.533564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.97.1:52853)
0926 03:00:15.533767 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.533866 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 03:00:15.538100 7323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:15.538439 7323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:15.545827 7323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb9850d450f64613a023c5b3e841da85: Sending DeleteTablet for 3 replicas of tablet 3574bf49e149450789101db0c01cc3a2
I20260926 03:00:15.546698 7438 tablet_service.cc:1514] Processing DeleteTablet for tablet 3574bf49e149450789101db0c01cc3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:15 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:51490
I20260926 03:00:15.546900 7583 tablet_service.cc:1514] Processing DeleteTablet for tablet 3574bf49e149450789101db0c01cc3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:15 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50260
I20260926 03:00:15.548534 7725 tablet_service.cc:1514] Processing DeleteTablet for tablet 3574bf49e149450789101db0c01cc3a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:15 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:50956
I20260926 03:00:15.550415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 7296
I20260926 03:00:15.557438 7914 tablet_replica.cc:330] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: stopping tablet replica
I20260926 03:00:15.557629 7915 tablet_replica.cc:330] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: stopping tablet replica
I20260926 03:00:15.557798 7915 raft_consensus.cc:2229] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:15.557796 7914 raft_consensus.cc:2229] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:00:15.558081 7915 raft_consensus.cc:2258] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:15.558555 7914 raft_consensus.cc:2258] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:15.559041 7915 ts_tablet_manager.cc:1905] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:15.561641 7914 ts_tablet_manager.cc:1905] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:15.565481 7915 ts_tablet_manager.cc:1918] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:15.565611 7915 log.cc:1196] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/3574bf49e149450789101db0c01cc3a2
I20260926 03:00:15.566082 7913 tablet_replica.cc:330] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: stopping tablet replica
I20260926 03:00:15.566144 7915 ts_tablet_manager.cc:1939] T 3574bf49e149450789101db0c01cc3a2 P 5cb395bdd48542e1baa56277b11090f2: Deleting consensus metadata
I20260926 03:00:15.566293 7913 raft_consensus.cc:2229] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:15.566604 7913 raft_consensus.cc:2258] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:15.567198 7311 catalog_manager.cc:4841] TS 5cb395bdd48542e1baa56277b11090f2 (127.0.97.3:44007): tablet 3574bf49e149450789101db0c01cc3a2 (table test-table [id=a37d5899611f424fb5b807c499a3f189]) successfully deleted
I20260926 03:00:15.571635 7914 ts_tablet_manager.cc:1918] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:15.571785 7914 log.cc:1196] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/3574bf49e149450789101db0c01cc3a2
I20260926 03:00:15.572253 7914 ts_tablet_manager.cc:1939] T 3574bf49e149450789101db0c01cc3a2 P bd03a6db3c494566b954e4a9bc73d704: Deleting consensus metadata
I20260926 03:00:15.572681 7913 ts_tablet_manager.cc:1905] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:15.573464 7308 catalog_manager.cc:4841] TS bd03a6db3c494566b954e4a9bc73d704 (127.0.97.2:44699): tablet 3574bf49e149450789101db0c01cc3a2 (table test-table [id=a37d5899611f424fb5b807c499a3f189]) successfully deleted
I20260926 03:00:15.578928 7913 ts_tablet_manager.cc:1918] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:15.579088 7913 log.cc:1196] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/3574bf49e149450789101db0c01cc3a2
I20260926 03:00:15.579597 7913 ts_tablet_manager.cc:1939] T 3574bf49e149450789101db0c01cc3a2 P fd226c00ab9141e998c7d13428d6b268: Deleting consensus metadata
I20260926 03:00:15.580732 7309 catalog_manager.cc:4841] TS fd226c00ab9141e998c7d13428d6b268 (127.0.97.1:39395): tablet 3574bf49e149450789101db0c01cc3a2 (table test-table [id=a37d5899611f424fb5b807c499a3f189]) successfully deleted
I20260926 03:00:15.581506 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.551482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.0.97.1:39395)
0926 03:00:15.551644 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.551650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.565468 (+ 13818us) server_negotiation.cc:407] Connection header received
0926 03:00:15.565670 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.565677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.565776 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.565930 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:15.568254 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.569331 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.570481 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.571107 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.571214 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.574321 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:15.574336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:15.574355 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:15.574390 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:15.578396 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:15.579255 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:15.579267 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:15.579284 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:15.579443 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:15.579867 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:15.579871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:15.579874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:15.580334 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:15.580608 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.581032 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.581201 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:00:15.582176 7498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:15.697913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 7296
I20260926 03:00:15.719822 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 7515
I20260926 03:00:15.735424 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.720594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.0.97.2:44699)
0926 03:00:15.720782 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.720786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.720883 (+ 97us) server_negotiation.cc:407] Connection header received
0926 03:00:15.721110 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.721113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.721176 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.721259 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:15.722893 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.723664 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.725555 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.725944 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.726056 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.729350 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:15.729365 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:15.729388 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:15.729418 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:15.732667 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:15.733485 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:15.733491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:15.733493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:15.733673 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:15.734065 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:15.734068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:15.734070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:15.734450 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:15.734643 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.734956 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.735094 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:00:15.736189 7643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:15.834188 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.833484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46640)
0926 03:00:15.833911 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.834031 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":192,"threads_started":1}
I20260926 03:00:15.835862 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.835549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42493 (local address 127.0.0.1:46646)
0926 03:00:15.835687 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:15.835756 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 03:00:15.853871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 7515
I20260926 03:00:15.873301 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 7661
I20260926 03:00:15.885654 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:15.874065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.0.97.3:44007)
0926 03:00:15.874234 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:15.874238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:15.874256 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:00:15.874409 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:15.874413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:15.874485 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:15.874571 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:15.875848 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.876573 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.877816 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:15.878190 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:15.878281 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:15.880658 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:15.880670 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:15.880686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:15.880712 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:15.883335 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:15.883925 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:15.883932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:15.883934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:15.884047 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:15.884374 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:15.884377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:15.884379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:15.884687 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:15.884898 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:15.885219 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:00:15.885423 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:00:15.886286 7785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:15.976420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 7661
I20260926 03:00:15.993353 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 6998
I20260926 03:00:15.994319 7164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:16.110693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 6998
I20260926 03:00:16.130891 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 7097
I20260926 03:00:16.131906 7354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:16.249195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 7097
2026-09-26T03:00:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9485 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:00:16 dist-test-slave-fccw krb5kdc[7926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:00:16 dist-test-slave-fccw krb5kdc[7926](info): set up 2 sockets
Sep 26 03:00:16 dist-test-slave-fccw krb5kdc[7926](info): commencing operation
krb5kdc: starting...
W20260926 03:00:17.334110 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:00:17 dist-test-slave-fccw krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260926 03:00:17.532568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37443
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:17.846904 7939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:17.847169 7939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:17.847254 7939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:17.854080 7939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:17.854180 7939 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:00:17.854219 7939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:17.854279 7939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:17.854327 7939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:17.854389 7939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:17.863732 7939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37443
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:17.864886 7939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:17.866148 7939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:17.877134 7944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:17.877099 7945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:17.877193 7947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:17.877969 7939 server_base.cc:1034] running on GCE node
I20260926 03:00:17.878646 7939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:17.879910 7939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:17.881105 7939 hybrid_clock.cc:648] HybridClock initialized: now 1790391617881055 us; error 46 us; skew 500 ppm
I20260926 03:00:17.881449 7939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:17 dist-test-slave-fccw krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:17.884297 7939 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260926 03:00:17.885722 7939 webserver.cc:460] Webserver started at http://127.0.97.62:45689/ using document root <none> and password file <none>
I20260926 03:00:17.886390 7939 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:17.886518 7939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:17.886837 7939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:17.890095 7939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "da1db226618346f59d834ea832eeb41d"
format_stamp: "Formatted at 2026-09-26 03:00:17 on dist-test-slave-fccw"
server_key: "19187ed56aaa0e7ed203689d4b157ef2"
server_key_iv: "35f3236e5d2ba0a8d225bfce3639a0fb"
server_key_version: "encryptionkey@0"
I20260926 03:00:17.890865 7939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "da1db226618346f59d834ea832eeb41d"
format_stamp: "Formatted at 2026-09-26 03:00:17 on dist-test-slave-fccw"
server_key: "19187ed56aaa0e7ed203689d4b157ef2"
server_key_iv: "35f3236e5d2ba0a8d225bfce3639a0fb"
server_key_version: "encryptionkey@0"
I20260926 03:00:17.895637 7939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:00:17.898963 7954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:17.900208 7939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:00:17.900385 7939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "da1db226618346f59d834ea832eeb41d"
format_stamp: "Formatted at 2026-09-26 03:00:17 on dist-test-slave-fccw"
server_key: "19187ed56aaa0e7ed203689d4b157ef2"
server_key_iv: "35f3236e5d2ba0a8d225bfce3639a0fb"
server_key_version: "encryptionkey@0"
I20260926 03:00:17.900595 7939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:17.911657 7939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:17.915268 7939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:17.915516 7939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:17.935966 7939 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37443
I20260926 03:00:17.935973 8015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37443 every 8 connection(s)
I20260926 03:00:17.937307 7939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:17.941213 8016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:17.942004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 7939
I20260926 03:00:17.942205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:17.942932 388 external_mini_cluster.cc:1428] Setting key 333254ff40802454f82942b7613f54d8
I20260926 03:00:17.954975 8016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Bootstrap starting.
Sep 26 03:00:17 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:17.957958 8016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:17.959316 8016 log.cc:824] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:17.962522 8016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: No bootstrap required, opened a new log
I20260926 03:00:17.966001 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:17.944807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.97.62:37443)
0926 03:00:17.945696 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:17.945711 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:17.945772 (+ 61us) server_negotiation.cc:407] Connection header received
0926 03:00:17.947662 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:17.947689 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:17.948205 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:17.948652 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:17.949942 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:17.951895 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:17.953173 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:17.954153 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:17.954336 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:17.957999 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:17.958009 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:17.958028 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:17.958119 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:17.961786 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:17.962404 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:17.962411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:17.962419 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:17.962632 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:17.962985 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:17.962992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:17.962996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:17.963620 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:17.963917 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:17.964244 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 03:00:17.964676 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":387,"threads_started":1}
I20260926 03:00:17.971007 8016 raft_consensus.cc:348] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1db226618346f59d834ea832eeb41d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37443 } }
I20260926 03:00:17.971444 8016 raft_consensus.cc:374] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:17.971555 8016 raft_consensus.cc:729] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1db226618346f59d834ea832eeb41d, State: Initialized, Role: FOLLOWER
I20260926 03:00:17.972131 8016 consensus_queue.cc:260] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1db226618346f59d834ea832eeb41d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37443 } }
I20260926 03:00:17.972332 8016 raft_consensus.cc:388] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:17.972420 8016 raft_consensus.cc:482] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:17.972529 8016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:17.974485 8016 raft_consensus.cc:504] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1db226618346f59d834ea832eeb41d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37443 } }
I20260926 03:00:17.974912 8016 leader_election.cc:302] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da1db226618346f59d834ea832eeb41d; no voters:
I20260926 03:00:17.975477 8016 leader_election.cc:288] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:17.975580 8021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:17.976255 8021 raft_consensus.cc:686] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [term 1 LEADER]: Becoming Leader. State: Replica: da1db226618346f59d834ea832eeb41d, State: Running, Role: LEADER
I20260926 03:00:17.976854 8021 consensus_queue.cc:237] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1db226618346f59d834ea832eeb41d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37443 } }
I20260926 03:00:17.977519 8016 sys_catalog.cc:564] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:17.979714 8022 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da1db226618346f59d834ea832eeb41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1db226618346f59d834ea832eeb41d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37443 } } }
I20260926 03:00:17.979749 8023 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader da1db226618346f59d834ea832eeb41d. Latest consensus state: current_term: 1 leader_uuid: "da1db226618346f59d834ea832eeb41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1db226618346f59d834ea832eeb41d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37443 } } }
I20260926 03:00:17.980144 8022 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:17.980144 8023 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:17.981083 8028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:17.986543 8028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:17.995031 8028 catalog_manager.cc:1347] Generated new cluster ID: 23974f7189bf4455ae0776ed295d8ed0
I20260926 03:00:17.995146 8028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:18.033531 8028 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:18.035360 8028 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:18.042651 8028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Generated new TSK 0
I20260926 03:00:18.043525 8028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260926 03:00:18.106271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37443
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:18.430133 8042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:18.430392 8042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:18.430485 8042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:18.437831 8042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:18.437965 8042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:18.438124 8042 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:00:18.438258 8042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:18.447930 8042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:18.449203 8042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:18.450495 8042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:18.463557 8047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:18.463514 8048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:18.463591 8050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:18.464579 8042 server_base.cc:1034] running on GCE node
I20260926 03:00:18.465229 8042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:18.466116 8042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:18.467347 8042 hybrid_clock.cc:648] HybridClock initialized: now 1790391618467275 us; error 49 us; skew 500 ppm
I20260926 03:00:18.467631 8042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:18 dist-test-slave-fccw krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:18.470626 8042 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:00:18.472018 8042 webserver.cc:460] Webserver started at http://127.0.97.1:32839/ using document root <none> and password file <none>
I20260926 03:00:18.472693 8042 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:18.472820 8042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:18.473196 8042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:18.475914 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "f0d99b9c9e454bd7b77f67f25bf45214"
format_stamp: "Formatted at 2026-09-26 03:00:18 on dist-test-slave-fccw"
server_key: "269dcd51adda6280a5f5d6448919f29c"
server_key_iv: "91e9518097378e9468a9745df93fc5f4"
server_key_version: "encryptionkey@0"
I20260926 03:00:18.476693 8042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f0d99b9c9e454bd7b77f67f25bf45214"
format_stamp: "Formatted at 2026-09-26 03:00:18 on dist-test-slave-fccw"
server_key: "269dcd51adda6280a5f5d6448919f29c"
server_key_iv: "91e9518097378e9468a9745df93fc5f4"
server_key_version: "encryptionkey@0"
I20260926 03:00:18.481691 8042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:00:18.485054 8057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:18.486302 8042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:00:18.486516 8042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "f0d99b9c9e454bd7b77f67f25bf45214"
format_stamp: "Formatted at 2026-09-26 03:00:18 on dist-test-slave-fccw"
server_key: "269dcd51adda6280a5f5d6448919f29c"
server_key_iv: "91e9518097378e9468a9745df93fc5f4"
server_key_version: "encryptionkey@0"
I20260926 03:00:18.486760 8042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:18.519465 8042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:18.523452 8042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:18.523748 8042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:18.526458 8042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:18.526574 8042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:18.526746 8042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:18.526852 8042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 26 03:00:18 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:18.549135 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:18.526954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52267 (local address 127.0.97.62:37443)
0926 03:00:18.527912 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:18.527919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:18.528785 (+ 866us) server_negotiation.cc:407] Connection header received
0926 03:00:18.530775 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:18.530783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:18.530879 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:18.531028 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:18.533437 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.534504 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.536434 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.537247 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.537408 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:18.541886 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:18.541891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:18.541894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:18.541925 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:18.545375 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:18.546345 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:18.546351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:18.546354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:18.546481 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:18.547087 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:18.547092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:18.547095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:18.547445 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:18.547620 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:18.548651 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0926 03:00:18.548819 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":737,"threads_started":1}
I20260926 03:00:18.551128 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:18.527733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37443 (local address 127.0.97.1:52267)
0926 03:00:18.528601 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:18.528644 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:18.530507 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:18.531343 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:18.531354 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:18.532214 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:18.533209 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:18.533238 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.534662 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.534667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:18.536257 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:18.536280 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.537517 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.537526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:18.537894 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:18.538655 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:18.538694 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:18.541633 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:18.545546 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:18.545555 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:18.545570 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:18.546214 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:18.546599 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:18.546604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:18.546607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:18.546928 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:18.547611 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:18.547623 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:18.548196 (+ 573us) client_negotiation.cc:769] Sending connection context
0926 03:00:18.548575 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 03:00:18.549295 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":352,"threads_started":1}
I20260926 03:00:18.575131 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:18.559918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60367 (local address 127.0.97.62:37443)
0926 03:00:18.560128 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:18.560137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:18.560707 (+ 570us) server_negotiation.cc:407] Connection header received
0926 03:00:18.560826 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:18.560833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:18.560929 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:18.561093 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:18.561926 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.563319 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.565406 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.565987 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.566136 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:18.568819 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:18.568827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:18.568834 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:18.568873 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:18.571924 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:18.572837 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:18.572845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:18.572851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:18.573004 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:18.573496 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:18.573503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:18.573509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:18.573914 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:18.574071 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:18.574576 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 03:00:18.574784 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:00:18.575255 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:18.559916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37443 (local address 127.0.97.1:60367)
0926 03:00:18.560608 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:18.560629 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:18.560734 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:18.561107 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:18.561110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:18.561517 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:18.561783 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:18.561791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.563448 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.563451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:18.565210 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:18.565222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:18.566229 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:18.566238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:18.566481 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:18.567175 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:18.567206 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:18.568601 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:18.572070 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:18.572076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:18.572080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:18.572647 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:18.573142 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:18.573147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:18.573153 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:18.573358 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:18.574077 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:18.574083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:18.574308 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 03:00:18.574479 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 03:00:18.574808 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":381,"threads_started":1}
I20260926 03:00:18.581718 8042 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35775
I20260926 03:00:18.581740 8177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35775 every 8 connection(s)
I20260926 03:00:18.582940 8042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:18.584880 8178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37443
I20260926 03:00:18.585143 8178 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:18.585760 8178 heartbeater.cc:511] Master 127.0.97.62:37443 requested a full tablet report, sending...
I20260926 03:00:18.587581 7971 ts_manager.cc:194] Registered new tserver with Master: f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775)
I20260926 03:00:18.588703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 8042
I20260926 03:00:18.588891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:00:18.589504 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52267
I20260926 03:00:18.589680 388 external_mini_cluster.cc:1428] Setting key 0cb7e77b87f048aa8fdffc6ea333d8b6
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260926 03:00:18.632841 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37443
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:18.950814 8182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:18.951067 8182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:18.951223 8182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:18.958551 8182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:18.958696 8182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:18.958847 8182 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:00:18.958971 8182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:18.968544 8182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:18.969836 8182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:18.971212 8182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:18.985183 8187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:18.985296 8188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:18.985348 8190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:18.986557 8182 server_base.cc:1034] running on GCE node
I20260926 03:00:18.987047 8182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:18.987958 8182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:18.989151 8182 hybrid_clock.cc:648] HybridClock initialized: now 1790391618989105 us; error 41 us; skew 500 ppm
I20260926 03:00:18.989434 8182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:18 dist-test-slave-fccw krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:18.992660 8182 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:00:18.993964 8182 webserver.cc:460] Webserver started at http://127.0.97.2:41907/ using document root <none> and password file <none>
I20260926 03:00:18.994552 8182 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:18.994633 8182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:18.994966 8182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:18.997792 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "962a1e07dcba40eb9d6e8d4a87b55d67"
format_stamp: "Formatted at 2026-09-26 03:00:18 on dist-test-slave-fccw"
server_key: "eef94b658b491ffd0d6700270b3aaa14"
server_key_iv: "7d2d3b45779f67c7863b370b247bf29d"
server_key_version: "encryptionkey@0"
I20260926 03:00:18.998512 8182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "962a1e07dcba40eb9d6e8d4a87b55d67"
format_stamp: "Formatted at 2026-09-26 03:00:18 on dist-test-slave-fccw"
server_key: "eef94b658b491ffd0d6700270b3aaa14"
server_key_iv: "7d2d3b45779f67c7863b370b247bf29d"
server_key_version: "encryptionkey@0"
I20260926 03:00:19.003533 8182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:00:19.006757 8197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:19.007956 8182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:00:19.008119 8182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "962a1e07dcba40eb9d6e8d4a87b55d67"
format_stamp: "Formatted at 2026-09-26 03:00:18 on dist-test-slave-fccw"
server_key: "eef94b658b491ffd0d6700270b3aaa14"
server_key_iv: "7d2d3b45779f67c7863b370b247bf29d"
server_key_version: "encryptionkey@0"
I20260926 03:00:19.008327 8182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:19.043489 8182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:19.047433 8182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:19.047703 8182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:19.050429 8182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:19.050531 8182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:19.050669 8182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:19.050726 8182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:00:19 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:19.074009 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.050797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40013 (local address 127.0.97.62:37443)
0926 03:00:19.051023 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.051031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.052437 (+ 1406us) server_negotiation.cc:407] Connection header received
0926 03:00:19.054698 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.054710 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.054816 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.054931 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:19.057371 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.058412 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.060873 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.061524 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.061647 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.066893 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:19.066898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:19.066902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:19.066930 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:19.070245 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.071139 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.071147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.071152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.071286 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.071888 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.071892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.071894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.072270 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:19.072486 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.073440 (+ 954us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.073652 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:00:19.075098 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.051467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37443 (local address 127.0.97.2:40013)
0926 03:00:19.052289 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.052325 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.054402 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.055218 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.055234 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.056135 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:19.057169 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.057204 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.058540 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.058551 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.060713 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.060729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.062005 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.062015 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.062296 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.063126 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:19.063170 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:19.066681 (+ 3511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:19.070389 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.070401 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.070419 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.070971 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.071403 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.071409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.071412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.071733 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.072550 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:19.072558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:19.073059 (+ 501us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.073357 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.073861 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":342,"threads_started":1}
I20260926 03:00:19.096246 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.080509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37443 (local address 127.0.97.2:46097)
0926 03:00:19.081214 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.081239 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.081392 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.081735 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.081741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.082388 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:19.082961 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.082982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.084376 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.084381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.086061 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.086075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.086835 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.086842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.087006 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.087662 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:19.087678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:19.088325 (+ 647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:19.091785 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.091791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.091795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.092207 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.092968 (+ 761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.092974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.092980 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.093255 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.094368 (+ 1113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:19.094375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:19.094691 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.094898 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.095405 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":347,"threads_started":1}
I20260926 03:00:19.097481 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.080409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46097 (local address 127.0.97.62:37443)
0926 03:00:19.080558 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.080561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.081340 (+ 779us) server_negotiation.cc:407] Connection header received
0926 03:00:19.081534 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.081539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.081604 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.081698 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:19.083098 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.084233 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.086215 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.086646 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.086787 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.088493 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:19.088499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:19.088503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:19.088540 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:19.091665 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.092386 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.092396 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.092402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.092644 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.093501 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.093509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.093514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.094141 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:19.096764 (+ 2623us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.096921 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.097162 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:00:19.102918 8182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39077
I20260926 03:00:19.102942 8316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39077 every 8 connection(s)
I20260926 03:00:19.104059 8182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:19.105885 8317 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37443
I20260926 03:00:19.106071 8317 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:19.106622 8317 heartbeater.cc:511] Master 127.0.97.62:37443 requested a full tablet report, sending...
I20260926 03:00:19.107750 7971 ts_manager.cc:194] Registered new tserver with Master: 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077)
I20260926 03:00:19.108495 7971 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40013
I20260926 03:00:19.113444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 8182
I20260926 03:00:19.113608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:00:19.114032 388 external_mini_cluster.cc:1428] Setting key c4d3614fa16335d7274d2a0d2110803e
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:00:19.158138 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37443
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:19.476754 8321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:19.477095 8321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:19.477270 8321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:19.484769 8321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:19.484867 8321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:19.484948 8321 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:00:19.485097 8321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:19.494967 8321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:19.496163 8321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:19.497429 8321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:19.510046 8329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:19.510072 8327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:19.510157 8326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:19.510890 8321 server_base.cc:1034] running on GCE node
I20260926 03:00:19.511509 8321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:19.512414 8321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:19.513648 8321 hybrid_clock.cc:648] HybridClock initialized: now 1790391619513561 us; error 65 us; skew 500 ppm
I20260926 03:00:19.513927 8321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:19 dist-test-slave-fccw krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:19.517072 8321 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:00:19.518460 8321 webserver.cc:460] Webserver started at http://127.0.97.3:43995/ using document root <none> and password file <none>
I20260926 03:00:19.519114 8321 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:19.519227 8321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:19.519547 8321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:19.522390 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "ae5a41317a5346bfa661ed4f148658b6"
format_stamp: "Formatted at 2026-09-26 03:00:19 on dist-test-slave-fccw"
server_key: "6a54a30f1a7f44faea1400aa8c895e3d"
server_key_iv: "0b9c26f1d7741f37a0f7671530fd541e"
server_key_version: "encryptionkey@0"
I20260926 03:00:19.523142 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ae5a41317a5346bfa661ed4f148658b6"
format_stamp: "Formatted at 2026-09-26 03:00:19 on dist-test-slave-fccw"
server_key: "6a54a30f1a7f44faea1400aa8c895e3d"
server_key_iv: "0b9c26f1d7741f37a0f7671530fd541e"
server_key_version: "encryptionkey@0"
I20260926 03:00:19.528003 8321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:00:19.531272 8336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:19.532569 8321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:00:19.532732 8321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "ae5a41317a5346bfa661ed4f148658b6"
format_stamp: "Formatted at 2026-09-26 03:00:19 on dist-test-slave-fccw"
server_key: "6a54a30f1a7f44faea1400aa8c895e3d"
server_key_iv: "0b9c26f1d7741f37a0f7671530fd541e"
server_key_version: "encryptionkey@0"
I20260926 03:00:19.532959 8321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:19.552426 8321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:19.556316 8321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:19.556596 8321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:19.559186 8321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:19.559269 8321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:19.559451 8321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:19.559507 8321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:00:19 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:19.584462 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.559956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38965 (local address 127.0.97.62:37443)
0926 03:00:19.560123 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.560130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.561560 (+ 1430us) server_negotiation.cc:407] Connection header received
0926 03:00:19.563592 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.563597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.563685 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.563791 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:19.566726 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.567700 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.570117 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.570894 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.571033 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.576034 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:19.576043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:19.576048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:19.576092 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:19.580321 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.581371 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.581376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.581379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.581546 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.582129 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.582134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.582137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.582531 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:19.582680 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.583851 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.584084 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:00:19.586153 8345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.560493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37443 (local address 127.0.97.3:38965)
0926 03:00:19.561374 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.561422 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.563314 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.564277 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.564293 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.565362 (+ 1069us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:19.566429 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.566457 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.567942 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.567953 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.569950 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.569968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.571180 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.571192 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.571460 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.572273 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:19.572305 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:19.575761 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:19.580503 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.580516 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.580540 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.581191 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.581673 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.581681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.581687 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.581984 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.582673 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:19.582684 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:19.583292 (+ 608us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.583677 (+ 385us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.584349 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":370,"threads_started":1}
I20260926 03:00:19.592772 8178 heartbeater.cc:503] Master 127.0.97.62:37443 was elected leader, sending a full tablet report...
I20260926 03:00:19.606736 8321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38313
I20260926 03:00:19.606740 8455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38313 every 8 connection(s)
I20260926 03:00:19.608008 8321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:19.609251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 8321
I20260926 03:00:19.609426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:00:19.609977 388 external_mini_cluster.cc:1428] Setting key 407e892530556ed0c03e2a80a6a37417
I20260926 03:00:19.610054 8456 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37443
I20260926 03:00:19.610265 8456 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:19.611092 8456 heartbeater.cc:511] Master 127.0.97.62:37443 requested a full tablet report, sending...
I20260926 03:00:19.612598 7970 ts_manager.cc:194] Registered new tserver with Master: ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313)
I20260926 03:00:19.613698 7970 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38965
I20260926 03:00:19.614483 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.597997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51969 (local address 127.0.97.62:37443)
0926 03:00:19.598177 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.598182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.598343 (+ 161us) server_negotiation.cc:407] Connection header received
0926 03:00:19.598579 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.598584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.598711 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.598811 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:19.600426 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.601747 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.603914 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.604538 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.604671 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.607172 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:19.607177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:19.607180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:19.607207 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:19.610988 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.612066 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.612074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.612080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.612283 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.612813 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.612820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.612825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.613262 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:19.613442 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.613948 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.614162 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:00:19.615386 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.597636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37443 (local address 127.0.97.3:51969)
0926 03:00:19.598249 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.598264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.598439 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.599011 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.599017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.599722 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:19.600229 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.600243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.601916 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.601926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.603689 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.603701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.604733 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.604739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.604916 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.605751 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:19.605780 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:19.607013 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:19.611176 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.611184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.611203 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.611899 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.612480 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:19.612487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:19.612494 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:19.612668 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:19.613407 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:19.613412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:19.613610 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.614652 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.614962 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":404,"threads_started":1}
I20260926 03:00:19.625128 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:00:19 dist-test-slave-fccw krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391619, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:00:19 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:19 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:00:19.670877 8066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:19.660983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.97.62:37443)
0926 03:00:19.661214 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.661218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.661508 (+ 290us) server_negotiation.cc:407] Connection header received
0926 03:00:19.661648 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.661652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.661730 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.661833 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:19.663083 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.663790 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.664924 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.665426 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.665518 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.670583 (+ 5065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58190: BlockingRecv error: recv got EOF from 127.0.0.1:58190 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 26 03:00:19 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391619, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:19.686067 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.673858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.0.97.62:37443)
0926 03:00:19.674005 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.674009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.674197 (+ 188us) server_negotiation.cc:407] Connection header received
0926 03:00:19.674312 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.674315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.674370 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.674458 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:19.675628 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.676319 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.677651 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.678047 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.678137 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.681570 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:19.681575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:19.681578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:19.681607 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:19.683957 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.684579 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.684582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.684584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.684692 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:19.685039 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:19.685050 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:19.685053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:19.685306 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:19.685503 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.685676 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.685838 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:00:19.689982 7970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:00:19.696877 7970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:19.729496 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.709641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.0.97.2:39077)
0926 03:00:19.722642 (+ 13001us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.722652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.722683 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:00:19.722771 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.722780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.723005 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.723203 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:19.723796 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.725110 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.727155 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.728405 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.728547 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.728751 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.728985 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.729153 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12843,"thread_start_us":12717,"threads_started":1}
I20260926 03:00:19.729568 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.708915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39077 (local address 127.0.0.1:49946)
0926 03:00:19.721285 (+ 12370us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.721306 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.721506 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.723219 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.723223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.723244 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:19.723610 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.723622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.725246 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.725251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.727006 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.727017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.728586 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.728592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.728773 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.728820 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.728948 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.729145 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12214,"spinlock_wait_cycles":1408,"thread_start_us":13762,"threads_started":1}
I20260926 03:00:19.730515 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.708870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.0.97.3:38313)
0926 03:00:19.720740 (+ 11870us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.720752 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.721687 (+ 935us) server_negotiation.cc:407] Connection header received
0926 03:00:19.721810 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.721818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.722065 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.722254 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:19.722978 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.725075 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.727806 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.729216 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.729334 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.729576 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.730070 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.730214 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11655,"thread_start_us":11443,"threads_started":1}
I20260926 03:00:19.730540 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.708615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38313 (local address 127.0.0.1:39618)
0926 03:00:19.721252 (+ 12637us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.721302 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.721506 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.722256 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.722263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.722289 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:19.722711 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.722724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.725204 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.725211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.727613 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.727638 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.729727 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.729733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.729895 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.729926 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.730016 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.730142 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12342,"mutex_wait_us":42,"spinlock_wait_cycles":3840,"thread_start_us":13032,"threads_started":1}
I20260926 03:00:19.731561 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.710081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.0.97.1:35775)
0926 03:00:19.722242 (+ 12161us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.722253 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.723038 (+ 785us) server_negotiation.cc:407] Connection header received
0926 03:00:19.723219 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.723227 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.723437 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.723630 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:19.724216 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.725650 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.728110 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.729538 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.730110 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.730303 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.731127 (+ 824us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.731274 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11991,"thread_start_us":11870,"threads_started":1}
I20260926 03:00:19.731571 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.709267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35775 (local address 127.0.0.1:43094)
0926 03:00:19.722927 (+ 13660us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.722946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.723065 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.723642 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.723650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.723674 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:19.724019 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.724030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.725788 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.725793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.727953 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.727966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.730445 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.730451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.730956 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.730979 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.731087 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.731202 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13546,"mutex_wait_us":1,"thread_start_us":11688,"threads_started":1}
I20260926 03:00:19.733141 8247 tablet_service.cc:1467] Processing CreateTablet for tablet df1abe2ec0c443c789c11db848f60780 (DEFAULT_TABLE table=test-table [id=55acdb4d6cd34a0d900f75b6248959b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:19.734568 8247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1abe2ec0c443c789c11db848f60780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:19.734823 8386 tablet_service.cc:1467] Processing CreateTablet for tablet df1abe2ec0c443c789c11db848f60780 (DEFAULT_TABLE table=test-table [id=55acdb4d6cd34a0d900f75b6248959b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:19.735291 8108 tablet_service.cc:1467] Processing CreateTablet for tablet df1abe2ec0c443c789c11db848f60780 (DEFAULT_TABLE table=test-table [id=55acdb4d6cd34a0d900f75b6248959b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:19.735886 8386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1abe2ec0c443c789c11db848f60780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:19.736440 8108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df1abe2ec0c443c789c11db848f60780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:19.746866 8480 tablet_bootstrap.cc:492] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Bootstrap starting.
I20260926 03:00:19.748548 8481 tablet_bootstrap.cc:492] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Bootstrap starting.
I20260926 03:00:19.749898 8482 tablet_bootstrap.cc:492] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Bootstrap starting.
I20260926 03:00:19.751539 8481 tablet_bootstrap.cc:654] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:19.751539 8480 tablet_bootstrap.cc:654] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:19.752456 8482 tablet_bootstrap.cc:654] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:19.752785 8480 log.cc:824] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:19.752800 8481 log.cc:824] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:19.753767 8482 log.cc:824] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:19.755587 8481 tablet_bootstrap.cc:492] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: No bootstrap required, opened a new log
I20260926 03:00:19.755752 8480 tablet_bootstrap.cc:492] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: No bootstrap required, opened a new log
I20260926 03:00:19.755877 8481 ts_tablet_manager.cc:1397] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 03:00:19.756000 8480 ts_tablet_manager.cc:1397] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:00:19.756480 8482 tablet_bootstrap.cc:492] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: No bootstrap required, opened a new log
I20260926 03:00:19.756726 8482 ts_tablet_manager.cc:1397] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 03:00:19.763015 8481 raft_consensus.cc:348] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.763355 8481 raft_consensus.cc:374] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:19.763451 8481 raft_consensus.cc:729] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962a1e07dcba40eb9d6e8d4a87b55d67, State: Initialized, Role: FOLLOWER
I20260926 03:00:19.764038 8481 consensus_queue.cc:260] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.764974 8317 heartbeater.cc:503] Master 127.0.97.62:37443 was elected leader, sending a full tablet report...
I20260926 03:00:19.766793 8481 ts_tablet_manager.cc:1428] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 03:00:19.766572 8480 raft_consensus.cc:348] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.766952 8480 raft_consensus.cc:374] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:19.767041 8480 raft_consensus.cc:729] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5a41317a5346bfa661ed4f148658b6, State: Initialized, Role: FOLLOWER
I20260926 03:00:19.767139 8482 raft_consensus.cc:348] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.767511 8482 raft_consensus.cc:374] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:19.767624 8482 raft_consensus.cc:729] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d99b9c9e454bd7b77f67f25bf45214, State: Initialized, Role: FOLLOWER
I20260926 03:00:19.767633 8480 consensus_queue.cc:260] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.768332 8482 consensus_queue.cc:260] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.770673 8482 ts_tablet_manager.cc:1428] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260926 03:00:19.771559 8456 heartbeater.cc:503] Master 127.0.97.62:37443 was elected leader, sending a full tablet report...
I20260926 03:00:19.772231 8480 ts_tablet_manager.cc:1428] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
W20260926 03:00:19.835176 8179 tablet.cc:2369] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:00:19.856139 8318 tablet.cc:2369] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:00:19.864439 8457 tablet.cc:2369] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:00:19.971953 8488 raft_consensus.cc:482] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:19.972280 8488 raft_consensus.cc:504] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.973871 8488 leader_election.cc:288] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313), 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077)
I20260926 03:00:19.980387 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.974414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49355 (local address 127.0.97.3:38313)
0926 03:00:19.974642 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.974650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.975094 (+ 444us) server_negotiation.cc:407] Connection header received
0926 03:00:19.975326 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.975330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.975410 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.975490 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:19.976020 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.976780 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.978762 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.979505 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.979665 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.979769 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.980016 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.980130 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:00:19.980451 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.974266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38313 (local address 127.0.97.1:49355)
0926 03:00:19.974983 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.974998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.975177 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.975515 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.975520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.975544 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:19.975873 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.975881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.976905 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.976912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.978602 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.978613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.979722 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.979728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.979869 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.979892 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.979993 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.980118 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":360,"threads_started":1}
I20260926 03:00:19.980841 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.974429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42949 (local address 127.0.97.2:39077)
0926 03:00:19.974799 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:19.974805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:19.975414 (+ 609us) server_negotiation.cc:407] Connection header received
0926 03:00:19.975578 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:19.975582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:19.975637 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:19.975743 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:19.976348 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.977091 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.979017 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.979938 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.980080 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.980184 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:19.980462 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 03:00:19.980604 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260926 03:00:19.981258 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:19.974429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39077 (local address 127.0.97.1:42949)
0926 03:00:19.975312 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:19.975326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:19.975433 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:19.975756 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:19.975760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:19.975796 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:19.976214 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.976223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.977215 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.977222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.978882 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:19.978893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:19.980156 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:19.980163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:19.980319 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:19.980340 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 03:00:19.980718 (+ 378us) client_negotiation.cc:240] Negotiation successful
0926 03:00:19.980860 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":629,"threads_started":1}
I20260926 03:00:19.981375 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1abe2ec0c443c789c11db848f60780" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5a41317a5346bfa661ed4f148658b6" is_pre_election: true
I20260926 03:00:19.981961 8406 raft_consensus.cc:2454] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 0.
I20260926 03:00:19.982241 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1abe2ec0c443c789c11db848f60780" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" is_pre_election: true
I20260926 03:00:19.982620 8059 leader_election.cc:302] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5a41317a5346bfa661ed4f148658b6, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:19.982663 8267 raft_consensus.cc:2454] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 0.
I20260926 03:00:19.983023 8488 raft_consensus.cc:2781] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:19.983240 8488 raft_consensus.cc:482] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:19.983369 8488 raft_consensus.cc:3037] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:19.985623 8488 raft_consensus.cc:504] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.986158 8488 leader_election.cc:288] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 election: Requested vote from peers ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313), 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077)
I20260926 03:00:19.986563 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1abe2ec0c443c789c11db848f60780" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5a41317a5346bfa661ed4f148658b6"
I20260926 03:00:19.986641 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df1abe2ec0c443c789c11db848f60780" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67"
I20260926 03:00:19.986740 8406 raft_consensus.cc:3037] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:19.986871 8267 raft_consensus.cc:3037] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:19.989058 8267 raft_consensus.cc:2454] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 1.
I20260926 03:00:19.989094 8406 raft_consensus.cc:2454] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 1.
I20260926 03:00:19.989477 8059 leader_election.cc:302] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5a41317a5346bfa661ed4f148658b6, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:19.989820 8488 raft_consensus.cc:2781] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:19.990252 8488 raft_consensus.cc:686] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 LEADER]: Becoming Leader. State: Replica: f0d99b9c9e454bd7b77f67f25bf45214, State: Running, Role: LEADER
I20260926 03:00:19.990712 8488 consensus_queue.cc:237] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } }
I20260926 03:00:19.995260 7970 catalog_manager.cc:5495] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } health_report { overall_health: HEALTHY } } }
I20260926 03:00:20.020756 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.016030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.0.97.1:35775)
0926 03:00:20.016206 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.016211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.016227 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:00:20.016378 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.016382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.016467 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.016582 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:20.016871 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.017552 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.018817 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.019177 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.019265 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.019675 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.020165 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.020268 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.020369 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.020427 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:00:20.028254 8406 raft_consensus.cc:1261] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Refusing update from remote peer f0d99b9c9e454bd7b77f67f25bf45214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.028244 8267 raft_consensus.cc:1261] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Refusing update from remote peer f0d99b9c9e454bd7b77f67f25bf45214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.029289 8491 consensus_queue.cc:1035] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.029676 8488 consensus_queue.cc:1035] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.041635 8495 mvcc.cc:205] Tried to move back new op lower bound from 7333444075627196416 to 7333444075485392896. Current Snapshot: MvccSnapshot[applied={T|T < 7333444075627196416}]
I20260926 03:00:20.043174 8494 mvcc.cc:205] Tried to move back new op lower bound from 7333444075627196416 to 7333444075485392896. Current Snapshot: MvccSnapshot[applied={T|T < 7333444075627196416}]
I20260926 03:00:20.050442 8496 mvcc.cc:205] Tried to move back new op lower bound from 7333444075627196416 to 7333444075485392896. Current Snapshot: MvccSnapshot[applied={T|T < 7333444075627196416}]
I20260926 03:00:20.056529 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:20.056768 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:20.060655 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Sending DeleteTablet for 3 replicas of tablet df1abe2ec0c443c789c11db848f60780
I20260926 03:00:20.061731 8247 tablet_service.cc:1514] Processing DeleteTablet for tablet df1abe2ec0c443c789c11db848f60780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260926 03:00:20.061738 8386 tablet_service.cc:1514] Processing DeleteTablet for tablet df1abe2ec0c443c789c11db848f60780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39618
I20260926 03:00:20.062366 8108 tablet_service.cc:1514] Processing DeleteTablet for tablet df1abe2ec0c443c789c11db848f60780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43094
I20260926 03:00:20.064385 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:00:20.065196 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:20.074353 8505 tablet_replica.cc:330] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: stopping tablet replica
I20260926 03:00:20.074882 8505 raft_consensus.cc:2229] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:20.075446 8505 raft_consensus.cc:2258] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.076442 8386 tablet_service.cc:1467] Processing CreateTablet for tablet 732a7b125f6945e99f6961ca52eff4f1 (DEFAULT_TABLE table=test-table [id=b2eb4922e89d44fd820248d7f8a908b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.076975 8386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732a7b125f6945e99f6961ca52eff4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.076435 8246 tablet_service.cc:1467] Processing CreateTablet for tablet 732a7b125f6945e99f6961ca52eff4f1 (DEFAULT_TABLE table=test-table [id=b2eb4922e89d44fd820248d7f8a908b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.078099 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732a7b125f6945e99f6961ca52eff4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.083848 8480 tablet_bootstrap.cc:492] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: Bootstrap starting.
I20260926 03:00:20.084607 8481 tablet_bootstrap.cc:492] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: Bootstrap starting.
I20260926 03:00:20.086133 8480 tablet_bootstrap.cc:654] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.086297 8481 tablet_bootstrap.cc:654] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.086869 8504 tablet_replica.cc:330] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: stopping tablet replica
I20260926 03:00:20.087143 8505 ts_tablet_manager.cc:1905] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.088816 8480 tablet_bootstrap.cc:492] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: No bootstrap required, opened a new log
I20260926 03:00:20.088954 8480 ts_tablet_manager.cc:1397] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 03:00:20.089891 8480 raft_consensus.cc:348] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.090097 8480 raft_consensus.cc:374] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.090202 8480 raft_consensus.cc:729] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5a41317a5346bfa661ed4f148658b6, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.090585 8480 consensus_queue.cc:260] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.091213 8107 tablet_service.cc:1467] Processing CreateTablet for tablet 732a7b125f6945e99f6961ca52eff4f1 (DEFAULT_TABLE table=test-table [id=b2eb4922e89d44fd820248d7f8a908b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.091568 8480 ts_tablet_manager.cc:1428] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:00:20.091715 8107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 732a7b125f6945e99f6961ca52eff4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.092229 8504 raft_consensus.cc:2229] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:20.092689 8504 raft_consensus.cc:2258] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.095297 8504 ts_tablet_manager.cc:1905] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.098297 8482 tablet_bootstrap.cc:492] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: Bootstrap starting.
I20260926 03:00:20.098575 8506 tablet_replica.cc:330] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: stopping tablet replica
I20260926 03:00:20.098984 8506 raft_consensus.cc:2229] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:00:20.099665 8506 raft_consensus.cc:2258] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.099958 8482 tablet_bootstrap.cc:654] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.101560 8506 ts_tablet_manager.cc:1905] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.102474 8504 ts_tablet_manager.cc:1918] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:20.102640 8504 log.cc:1196] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/df1abe2ec0c443c789c11db848f60780
I20260926 03:00:20.102741 8481 tablet_bootstrap.cc:492] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: No bootstrap required, opened a new log
I20260926 03:00:20.102895 8481 ts_tablet_manager.cc:1397] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260926 03:00:20.103195 8504 ts_tablet_manager.cc:1939] T df1abe2ec0c443c789c11db848f60780 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting consensus metadata
I20260926 03:00:20.103741 8481 raft_consensus.cc:348] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.103958 8481 raft_consensus.cc:374] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.104060 8481 raft_consensus.cc:729] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962a1e07dcba40eb9d6e8d4a87b55d67, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.105227 7958 catalog_manager.cc:4841] TS 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077): tablet df1abe2ec0c443c789c11db848f60780 (table test-table [id=55acdb4d6cd34a0d900f75b6248959b2]) successfully deleted
I20260926 03:00:20.105376 8481 consensus_queue.cc:260] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.106355 8482 tablet_bootstrap.cc:492] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: No bootstrap required, opened a new log
I20260926 03:00:20.106501 8482 ts_tablet_manager.cc:1397] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260926 03:00:20.107261 8482 raft_consensus.cc:348] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.107523 8482 raft_consensus.cc:374] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.107645 8482 raft_consensus.cc:729] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d99b9c9e454bd7b77f67f25bf45214, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.107933 8482 consensus_queue.cc:260] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.108096 8481 ts_tablet_manager.cc:1428] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 03:00:20.109078 8482 ts_tablet_manager.cc:1428] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 03:00:20.109486 8506 ts_tablet_manager.cc:1918] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:20.109616 8506 log.cc:1196] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/df1abe2ec0c443c789c11db848f60780
I20260926 03:00:20.110067 8506 ts_tablet_manager.cc:1939] T df1abe2ec0c443c789c11db848f60780 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting consensus metadata
I20260926 03:00:20.111405 7955 catalog_manager.cc:4841] TS f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775): tablet df1abe2ec0c443c789c11db848f60780 (table test-table [id=55acdb4d6cd34a0d900f75b6248959b2]) successfully deleted
I20260926 03:00:20.114032 8505 ts_tablet_manager.cc:1918] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:20.114197 8505 log.cc:1196] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/df1abe2ec0c443c789c11db848f60780
I20260926 03:00:20.114657 8505 ts_tablet_manager.cc:1939] T df1abe2ec0c443c789c11db848f60780 P ae5a41317a5346bfa661ed4f148658b6: Deleting consensus metadata
I20260926 03:00:20.116197 7956 catalog_manager.cc:4841] TS ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313): tablet df1abe2ec0c443c789c11db848f60780 (table test-table [id=55acdb4d6cd34a0d900f75b6248959b2]) successfully deleted
I20260926 03:00:20.143998 8487 raft_consensus.cc:482] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:20.144208 8487 raft_consensus.cc:504] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.145902 8487 leader_election.cc:288] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775), 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077)
I20260926 03:00:20.153399 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.146440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43331 (local address 127.0.97.2:39077)
0926 03:00:20.146739 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.146746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.147803 (+ 1057us) server_negotiation.cc:407] Connection header received
0926 03:00:20.147932 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.147938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.148009 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.148095 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:20.148599 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.149695 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.151703 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.152422 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.152556 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.152643 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.152972 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.153122 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260926 03:00:20.153457 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.146320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39077 (local address 127.0.97.3:43331)
0926 03:00:20.147676 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:20.147704 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:20.147815 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:20.148113 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:20.148118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:20.148147 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:20.148438 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.148449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.149833 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.149838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.151569 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.151578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.152570 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.152576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.152763 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.152806 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:00:20.152943 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 03:00:20.153095 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"mutex_wait_us":1,"thread_start_us":1106,"threads_started":1}
I20260926 03:00:20.153676 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.146376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56507 (local address 127.0.97.1:35775)
0926 03:00:20.146584 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.146592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.147041 (+ 449us) server_negotiation.cc:407] Connection header received
0926 03:00:20.147373 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.147381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.147489 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.147589 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:20.148083 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.149072 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.151158 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.152402 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.152533 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.152635 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.153227 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.153373 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:00:20.154284 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a7b125f6945e99f6961ca52eff4f1" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" is_pre_election: true
I20260926 03:00:20.154448 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.146073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35775 (local address 127.0.97.3:56507)
0926 03:00:20.146928 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:20.146953 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:20.147116 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:20.147634 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:20.147639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:20.147669 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:20.147931 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.147948 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.149193 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.149198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.151000 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.151014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.152810 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.152817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.153049 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.153083 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:00:20.153929 (+ 846us) client_negotiation.cc:240] Negotiation successful
0926 03:00:20.154070 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":24,"thread_start_us":1034,"threads_started":1}
I20260926 03:00:20.154526 8267 raft_consensus.cc:2454] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 0.
I20260926 03:00:20.154992 8340 leader_election.cc:302] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962a1e07dcba40eb9d6e8d4a87b55d67, ae5a41317a5346bfa661ed4f148658b6; no voters:
I20260926 03:00:20.155406 8487 raft_consensus.cc:2781] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:20.155221 8129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a7b125f6945e99f6961ca52eff4f1" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" is_pre_election: true
I20260926 03:00:20.155524 8487 raft_consensus.cc:482] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:20.155603 8487 raft_consensus.cc:3037] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.155686 8129 raft_consensus.cc:2454] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 0.
I20260926 03:00:20.157953 8487 raft_consensus.cc:504] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.158595 8487 leader_election.cc:288] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 election: Requested vote from peers f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775), 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077)
I20260926 03:00:20.158932 8129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a7b125f6945e99f6961ca52eff4f1" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d99b9c9e454bd7b77f67f25bf45214"
I20260926 03:00:20.159122 8129 raft_consensus.cc:3037] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.159081 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "732a7b125f6945e99f6961ca52eff4f1" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67"
I20260926 03:00:20.159282 8267 raft_consensus.cc:3037] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.161405 8129 raft_consensus.cc:2454] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 1.
I20260926 03:00:20.161795 8337 leader_election.cc:302] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5a41317a5346bfa661ed4f148658b6, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:20.162060 8487 raft_consensus.cc:2781] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:20.162204 8267 raft_consensus.cc:2454] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 1.
I20260926 03:00:20.162890 8487 raft_consensus.cc:686] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 1 LEADER]: Becoming Leader. State: Replica: ae5a41317a5346bfa661ed4f148658b6, State: Running, Role: LEADER
I20260926 03:00:20.163364 8487 consensus_queue.cc:237] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.166625 7971 catalog_manager.cc:5495] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } health_report { overall_health: HEALTHY } } }
Sep 26 03:00:20 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:20.225065 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.207289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37443 (local address 127.0.0.1:58204)
0926 03:00:20.208000 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:20.208031 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:20.208191 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:20.208566 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:20.208570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:20.209366 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:20.209585 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.209593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.210664 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.210670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.211637 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.211648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.212391 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.212396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.212563 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.217403 (+ 4840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:20.217419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:20.219724 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:20.222656 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:20.222669 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:20.222684 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:20.223062 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:20.223407 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:20.223413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:20.223417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:20.223545 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:20.224033 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:20.224040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:20.224188 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 03:00:20.224338 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 03:00:20.224599 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":348,"threads_started":1}
I20260926 03:00:20.225097 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.207331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.97.62:37443)
0926 03:00:20.208337 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.208340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.208357 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:00:20.208391 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.208394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.208439 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.208545 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:20.209762 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.210532 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.211777 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.212223 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.212369 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.219921 (+ 7552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:20.219926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:20.219929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:20.219955 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:20.222533 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:20.223175 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:20.223189 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:20.223198 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:20.223306 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:20.223650 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:20.223655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:20.223658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:20.223910 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:20.224054 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.224420 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.224728 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":882,"thread_start_us":717,"threads_started":1}
I20260926 03:00:20.229761 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58204:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:00:20.230782 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:20.237289 8246 tablet_service.cc:1467] Processing CreateTablet for tablet 4f339db38634484dba2c4201aa508f5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0240ac0833c948eca5785131499801d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:00:20.237825 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f339db38634484dba2c4201aa508f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.237834 8386 tablet_service.cc:1467] Processing CreateTablet for tablet 4f339db38634484dba2c4201aa508f5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0240ac0833c948eca5785131499801d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:00:20.238376 8386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f339db38634484dba2c4201aa508f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.239476 8107 tablet_service.cc:1467] Processing CreateTablet for tablet 4f339db38634484dba2c4201aa508f5e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0240ac0833c948eca5785131499801d4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:00:20.240000 8107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f339db38634484dba2c4201aa508f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.244164 8481 tablet_bootstrap.cc:492] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: Bootstrap starting.
I20260926 03:00:20.245793 8481 tablet_bootstrap.cc:654] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.247047 8482 tablet_bootstrap.cc:492] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: Bootstrap starting.
I20260926 03:00:20.248802 8482 tablet_bootstrap.cc:654] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.249531 8480 tablet_bootstrap.cc:492] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: Bootstrap starting.
I20260926 03:00:20.251641 8480 tablet_bootstrap.cc:654] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.258706 8482 tablet_bootstrap.cc:492] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: No bootstrap required, opened a new log
I20260926 03:00:20.258915 8482 ts_tablet_manager.cc:1397] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260926 03:00:20.259927 8482 raft_consensus.cc:348] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.260248 8482 raft_consensus.cc:374] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.260349 8482 raft_consensus.cc:729] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d99b9c9e454bd7b77f67f25bf45214, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.260699 8482 consensus_queue.cc:260] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.261229 8488 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.261497 8488 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:00:20.261981 8482 ts_tablet_manager.cc:1428] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 03:00:20.262241 8480 tablet_bootstrap.cc:492] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: No bootstrap required, opened a new log
I20260926 03:00:20.262382 8480 ts_tablet_manager.cc:1397] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20260926 03:00:20.262733 8481 tablet_bootstrap.cc:492] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: No bootstrap required, opened a new log
I20260926 03:00:20.262871 8481 ts_tablet_manager.cc:1397] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.009s
I20260926 03:00:20.263552 8481 raft_consensus.cc:348] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.263799 8481 raft_consensus.cc:374] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.263868 8481 raft_consensus.cc:729] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962a1e07dcba40eb9d6e8d4a87b55d67, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.263087 8480 raft_consensus.cc:348] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.263988 8480 raft_consensus.cc:374] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.264055 8480 raft_consensus.cc:729] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5a41317a5346bfa661ed4f148658b6, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.264145 8481 consensus_queue.cc:260] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.264299 8480 consensus_queue.cc:260] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.264649 8487 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.264555 8486 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.264886 8486 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:00:20.264889 8487 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:00:20.265908 8481 ts_tablet_manager.cc:1428] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:00:20.267127 8480 ts_tablet_manager.cc:1428] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260926 03:00:20.303579 8487 raft_consensus.cc:482] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:20.303787 8487 raft_consensus.cc:504] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.304437 8487 leader_election.cc:288] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775), 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077)
I20260926 03:00:20.304925 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f339db38634484dba2c4201aa508f5e" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" is_pre_election: true
I20260926 03:00:20.304945 8129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f339db38634484dba2c4201aa508f5e" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" is_pre_election: true
I20260926 03:00:20.305177 8267 raft_consensus.cc:2454] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 0.
I20260926 03:00:20.305234 8129 raft_consensus.cc:2454] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 0.
I20260926 03:00:20.305521 8340 leader_election.cc:302] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962a1e07dcba40eb9d6e8d4a87b55d67, ae5a41317a5346bfa661ed4f148658b6; no voters:
I20260926 03:00:20.305837 8487 raft_consensus.cc:2781] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:20.305933 8487 raft_consensus.cc:482] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:20.306025 8487 raft_consensus.cc:3037] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.307950 8487 raft_consensus.cc:504] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.308495 8487 leader_election.cc:288] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 election: Requested vote from peers f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775), 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077)
I20260926 03:00:20.308876 8129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f339db38634484dba2c4201aa508f5e" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d99b9c9e454bd7b77f67f25bf45214"
I20260926 03:00:20.308897 8267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f339db38634484dba2c4201aa508f5e" candidate_uuid: "ae5a41317a5346bfa661ed4f148658b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67"
I20260926 03:00:20.309108 8129 raft_consensus.cc:3037] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.309131 8267 raft_consensus.cc:3037] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.311184 8129 raft_consensus.cc:2454] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 1.
I20260926 03:00:20.311525 8337 leader_election.cc:302] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae5a41317a5346bfa661ed4f148658b6, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:20.311786 8487 raft_consensus.cc:2781] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:20.311862 8267 raft_consensus.cc:2454] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5a41317a5346bfa661ed4f148658b6 in term 1.
I20260926 03:00:20.311981 8487 raft_consensus.cc:686] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [term 1 LEADER]: Becoming Leader. State: Replica: ae5a41317a5346bfa661ed4f148658b6, State: Running, Role: LEADER
I20260926 03:00:20.312500 8487 consensus_queue.cc:237] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.314260 8512 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: TxnStatusTablet state changed. Reason: New leader ae5a41317a5346bfa661ed4f148658b6. Latest consensus state: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.314601 8512 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:00:20.315340 8525 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:00:20.315559 8525 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:00:20.315671 7971 catalog_manager.cc:5495] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } health_report { overall_health: HEALTHY } } }
I20260926 03:00:20.334272 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.327019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38313 (local address 127.0.0.1:39634)
0926 03:00:20.327332 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:20.327351 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:20.327490 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:20.328008 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:20.328015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:20.328026 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:00:20.328234 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.328247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.329805 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.329813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.331414 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.331429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.332453 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.332459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.332635 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.332704 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.333716 (+ 1012us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.333737 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 03:00:20.333827 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 03:00:20.333979 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 03:00:20.334285 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.327102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.0.97.3:38313)
0926 03:00:20.327281 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.327287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.327424 (+ 137us) server_negotiation.cc:407] Connection header received
0926 03:00:20.327689 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.327695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.327825 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.327966 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:20.328374 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.329661 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.331614 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.332263 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.332395 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.332900 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.333589 (+ 689us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.333729 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.333865 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.333952 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:00:20.339113 8267 raft_consensus.cc:1261] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Refusing update from remote peer ae5a41317a5346bfa661ed4f148658b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.339535 8129 raft_consensus.cc:1261] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Refusing update from remote peer ae5a41317a5346bfa661ed4f148658b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.340164 8487 consensus_queue.cc:1035] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.340524 8512 consensus_queue.cc:1035] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.351397 8488 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: TxnStatusTablet state changed. Reason: New leader ae5a41317a5346bfa661ed4f148658b6. Latest consensus state: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.351718 8488 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:00:20.352983 8486 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: TxnStatusTablet state changed. Reason: New leader ae5a41317a5346bfa661ed4f148658b6. Latest consensus state: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.353302 8486 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:00:20.355660 8487 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.355942 8487 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:00:20.356647 8487 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.356874 8487 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:00:20.358578 8488 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.358822 8488 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P f0d99b9c9e454bd7b77f67f25bf45214: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:00:20.360577 8486 tablet_replica.cc:439] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae5a41317a5346bfa661ed4f148658b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } } }
I20260926 03:00:20.360802 8486 tablet_replica.cc:442] T 4f339db38634484dba2c4201aa508f5e P 962a1e07dcba40eb9d6e8d4a87b55d67: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:00:20.372118 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.365220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.0.97.3:38313)
0926 03:00:20.365400 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.365406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.365940 (+ 534us) server_negotiation.cc:407] Connection header received
0926 03:00:20.366006 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.366012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.366117 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.366208 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:20.366713 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.367872 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.369902 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.370638 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.370770 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.371321 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.371451 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.371556 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.371709 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.371792 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:00:20.391721 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.383856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51181 (local address 127.0.97.3:38313)
0926 03:00:20.384081 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.384088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.384779 (+ 691us) server_negotiation.cc:407] Connection header received
0926 03:00:20.384982 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.384987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.385087 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.385187 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:20.385825 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.387017 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.389278 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.390005 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.390146 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.390713 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.390868 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.390966 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.391139 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.391233 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:00:20.391782 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.383914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38313 (local address 127.0.97.3:51181)
0926 03:00:20.384663 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:20.384682 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:20.384843 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:20.385210 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:20.385216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:20.385229 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:00:20.385648 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.385660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.387172 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.387178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.389106 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.389121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.390213 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.390220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.390448 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.390529 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.390994 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.391009 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 03:00:20.391107 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:00:20.391322 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":54,"thread_start_us":394,"threads_started":1}
I20260926 03:00:20.403244 8267 raft_consensus.cc:1261] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Refusing update from remote peer ae5a41317a5346bfa661ed4f148658b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.403455 8128 raft_consensus.cc:1261] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Refusing update from remote peer ae5a41317a5346bfa661ed4f148658b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.403944 8512 consensus_queue.cc:1035] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.404398 8512 consensus_queue.cc:1035] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.501542 7970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:20.501803 7970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:20.505364 7970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Sending DeleteTablet for 3 replicas of tablet 732a7b125f6945e99f6961ca52eff4f1
I20260926 03:00:20.506175 8107 tablet_service.cc:1514] Processing DeleteTablet for tablet 732a7b125f6945e99f6961ca52eff4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43094
I20260926 03:00:20.506475 8506 tablet_replica.cc:330] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: stopping tablet replica
I20260926 03:00:20.506703 8506 raft_consensus.cc:2229] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:20.506835 8246 tablet_service.cc:1514] Processing DeleteTablet for tablet 732a7b125f6945e99f6961ca52eff4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260926 03:00:20.506902 8506 raft_consensus.cc:2258] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.507145 8504 tablet_replica.cc:330] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: stopping tablet replica
I20260926 03:00:20.507359 8504 raft_consensus.cc:2229] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:20.507476 8385 tablet_service.cc:1514] Processing DeleteTablet for tablet 732a7b125f6945e99f6961ca52eff4f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39618
I20260926 03:00:20.507539 8504 raft_consensus.cc:2258] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.507715 8505 tablet_replica.cc:330] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: stopping tablet replica
I20260926 03:00:20.507897 8505 raft_consensus.cc:2229] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:00:20.508348 8505 raft_consensus.cc:2258] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.509640 8505 ts_tablet_manager.cc:1905] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.509660 8506 ts_tablet_manager.cc:1905] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.517375 8506 ts_tablet_manager.cc:1918] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:00:20.518584 8504 ts_tablet_manager.cc:1905] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.523305 8505 ts_tablet_manager.cc:1918] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:00:20.523448 8505 log.cc:1196] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/732a7b125f6945e99f6961ca52eff4f1
I20260926 03:00:20.523854 8505 ts_tablet_manager.cc:1939] T 732a7b125f6945e99f6961ca52eff4f1 P ae5a41317a5346bfa661ed4f148658b6: Deleting consensus metadata
I20260926 03:00:20.524899 7956 catalog_manager.cc:4841] TS ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313): tablet 732a7b125f6945e99f6961ca52eff4f1 (table test-table [id=b2eb4922e89d44fd820248d7f8a908b5]) successfully deleted
I20260926 03:00:20.525725 8504 ts_tablet_manager.cc:1918] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:00:20.525957 8504 log.cc:1196] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/732a7b125f6945e99f6961ca52eff4f1
I20260926 03:00:20.526460 8504 ts_tablet_manager.cc:1939] T 732a7b125f6945e99f6961ca52eff4f1 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting consensus metadata
I20260926 03:00:20.526664 8506 log.cc:1196] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/732a7b125f6945e99f6961ca52eff4f1
I20260926 03:00:20.527171 8506 ts_tablet_manager.cc:1939] T 732a7b125f6945e99f6961ca52eff4f1 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting consensus metadata
I20260926 03:00:20.528290 7958 catalog_manager.cc:4841] TS 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077): tablet 732a7b125f6945e99f6961ca52eff4f1 (table test-table [id=b2eb4922e89d44fd820248d7f8a908b5]) successfully deleted
I20260926 03:00:20.529739 7955 catalog_manager.cc:4841] TS f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775): tablet 732a7b125f6945e99f6961ca52eff4f1 (table test-table [id=b2eb4922e89d44fd820248d7f8a908b5]) successfully deleted
Sep 26 03:00:20 dist-test-slave-fccw krb5kdc[7926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:00:20 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:20 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:00:20.592495 8520 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:20.577805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.0.97.62:37443)
0926 03:00:20.577999 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.578005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.578718 (+ 713us) server_negotiation.cc:407] Connection header received
0926 03:00:20.579002 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.579009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.579095 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.579166 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:20.581099 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.582259 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.584374 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.585151 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.585267 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.592224 (+ 6957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58208: BlockingRecv error: recv got EOF from 127.0.0.1:58208 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 03:00:20 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260926 03:00:20.619215 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.601404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.0.97.62:37443)
0926 03:00:20.601608 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.601613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.601860 (+ 247us) server_negotiation.cc:407] Connection header received
0926 03:00:20.602022 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.602028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.602102 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.602201 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:20.603713 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.604836 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.606755 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.607393 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.607487 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.612389 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:20.612396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:20.612403 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:20.612487 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:20.615806 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:20.616875 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:20.616880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:20.616884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:20.617080 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:20.617546 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:20.617551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:20.617555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:20.617896 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:20.618111 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.618669 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.618873 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:00:20.624104 7970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:00:20.624776 7970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:20.631870 8107 tablet_service.cc:1467] Processing CreateTablet for tablet fcdf75aa10ab40f597adb110d0e2a668 (DEFAULT_TABLE table=test-table [id=3d7201e033b14c0891ab63d5d46f09d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.632287 8107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcdf75aa10ab40f597adb110d0e2a668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.632264 8385 tablet_service.cc:1467] Processing CreateTablet for tablet fcdf75aa10ab40f597adb110d0e2a668 (DEFAULT_TABLE table=test-table [id=3d7201e033b14c0891ab63d5d46f09d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.632774 8385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcdf75aa10ab40f597adb110d0e2a668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.636813 8246 tablet_service.cc:1467] Processing CreateTablet for tablet fcdf75aa10ab40f597adb110d0e2a668 (DEFAULT_TABLE table=test-table [id=3d7201e033b14c0891ab63d5d46f09d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.637233 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcdf75aa10ab40f597adb110d0e2a668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.639587 8480 tablet_bootstrap.cc:492] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: Bootstrap starting.
I20260926 03:00:20.641498 8480 tablet_bootstrap.cc:654] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.647768 8481 tablet_bootstrap.cc:492] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: Bootstrap starting.
I20260926 03:00:20.649619 8481 tablet_bootstrap.cc:654] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.650063 8482 tablet_bootstrap.cc:492] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: Bootstrap starting.
I20260926 03:00:20.650633 8480 tablet_bootstrap.cc:492] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: No bootstrap required, opened a new log
I20260926 03:00:20.650874 8480 ts_tablet_manager.cc:1397] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260926 03:00:20.651530 8482 tablet_bootstrap.cc:654] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.652011 8480 raft_consensus.cc:348] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.652446 8480 raft_consensus.cc:374] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.652608 8480 raft_consensus.cc:729] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5a41317a5346bfa661ed4f148658b6, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.653034 8480 consensus_queue.cc:260] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.657032 8480 ts_tablet_manager.cc:1428] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260926 03:00:20.662411 8482 tablet_bootstrap.cc:492] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: No bootstrap required, opened a new log
I20260926 03:00:20.662597 8482 ts_tablet_manager.cc:1397] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20260926 03:00:20.663470 8482 raft_consensus.cc:348] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.663717 8482 raft_consensus.cc:374] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.663810 8482 raft_consensus.cc:729] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d99b9c9e454bd7b77f67f25bf45214, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.664034 8481 tablet_bootstrap.cc:492] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: No bootstrap required, opened a new log
I20260926 03:00:20.664028 8482 consensus_queue.cc:260] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.664170 8481 ts_tablet_manager.cc:1397] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260926 03:00:20.664574 8482 ts_tablet_manager.cc:1428] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 03:00:20.664911 8481 raft_consensus.cc:348] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.665189 8481 raft_consensus.cc:374] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.665287 8481 raft_consensus.cc:729] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962a1e07dcba40eb9d6e8d4a87b55d67, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.665511 8481 consensus_queue.cc:260] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.666014 8481 ts_tablet_manager.cc:1428] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:00:20.724748 8488 raft_consensus.cc:482] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:20.724964 8488 raft_consensus.cc:504] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.725646 8488 leader_election.cc:288] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077), ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313)
I20260926 03:00:20.726240 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcdf75aa10ab40f597adb110d0e2a668" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5a41317a5346bfa661ed4f148658b6" is_pre_election: true
I20260926 03:00:20.726234 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcdf75aa10ab40f597adb110d0e2a668" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" is_pre_election: true
I20260926 03:00:20.726473 8266 raft_consensus.cc:2454] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 0.
I20260926 03:00:20.726493 8406 raft_consensus.cc:2454] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 0.
I20260926 03:00:20.726913 8061 leader_election.cc:302] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962a1e07dcba40eb9d6e8d4a87b55d67, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:20.727159 8488 raft_consensus.cc:2781] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:20.727296 8488 raft_consensus.cc:482] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:20.727387 8488 raft_consensus.cc:3037] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.729327 8488 raft_consensus.cc:504] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.729858 8488 leader_election.cc:288] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 election: Requested vote from peers 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077), ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313)
I20260926 03:00:20.730238 8266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcdf75aa10ab40f597adb110d0e2a668" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67"
I20260926 03:00:20.730440 8266 raft_consensus.cc:3037] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.730369 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcdf75aa10ab40f597adb110d0e2a668" candidate_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5a41317a5346bfa661ed4f148658b6"
I20260926 03:00:20.730572 8406 raft_consensus.cc:3037] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.733412 8266 raft_consensus.cc:2454] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 1.
I20260926 03:00:20.733683 8406 raft_consensus.cc:2454] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0d99b9c9e454bd7b77f67f25bf45214 in term 1.
I20260926 03:00:20.733804 8061 leader_election.cc:302] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962a1e07dcba40eb9d6e8d4a87b55d67, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:20.734079 8488 raft_consensus.cc:2781] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:20.734521 8488 raft_consensus.cc:686] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 LEADER]: Becoming Leader. State: Replica: f0d99b9c9e454bd7b77f67f25bf45214, State: Running, Role: LEADER
I20260926 03:00:20.734804 8488 consensus_queue.cc:237] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.737444 7971 catalog_manager.cc:5495] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } health_report { overall_health: UNKNOWN } } }
I20260926 03:00:20.772143 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.766744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.0.97.1:35775)
0926 03:00:20.767234 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.767240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.767262 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:00:20.767327 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.767332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.767412 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.767525 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:20.767880 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.768940 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.770271 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.770916 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.771043 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.771383 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.771513 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.771605 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.771711 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.771791 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":237,"threads_started":1}
I20260926 03:00:20.774871 8406 raft_consensus.cc:1261] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Refusing update from remote peer f0d99b9c9e454bd7b77f67f25bf45214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.774919 8266 raft_consensus.cc:1261] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Refusing update from remote peer f0d99b9c9e454bd7b77f67f25bf45214: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:20.775493 8488 consensus_queue.cc:1035] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.775858 8554 consensus_queue.cc:1035] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:20.788722 7969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:20.788971 7969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:20.794574 7969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Sending DeleteTablet for 3 replicas of tablet fcdf75aa10ab40f597adb110d0e2a668
I20260926 03:00:20.795724 8246 tablet_service.cc:1514] Processing DeleteTablet for tablet fcdf75aa10ab40f597adb110d0e2a668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260926 03:00:20.796015 8504 tablet_replica.cc:330] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: stopping tablet replica
I20260926 03:00:20.796015 8385 tablet_service.cc:1514] Processing DeleteTablet for tablet fcdf75aa10ab40f597adb110d0e2a668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39618
I20260926 03:00:20.796191 8504 raft_consensus.cc:2229] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:20.796283 8505 tablet_replica.cc:330] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: stopping tablet replica
I20260926 03:00:20.796419 8504 raft_consensus.cc:2258] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.796478 8505 raft_consensus.cc:2229] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:20.796538 8107 tablet_service.cc:1514] Processing DeleteTablet for tablet fcdf75aa10ab40f597adb110d0e2a668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43094
I20260926 03:00:20.796697 8505 raft_consensus.cc:2258] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:20.797012 8506 tablet_replica.cc:330] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: stopping tablet replica
I20260926 03:00:20.797159 8385 tablet_service.cc:1514] Processing DeleteTablet for tablet fcdf75aa10ab40f597adb110d0e2a668 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39618
I20260926 03:00:20.797031 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 03:00:20.797192 8506 raft_consensus.cc:2229] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:00:20.797483 8506 raft_consensus.cc:2258] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 03:00:20.797636 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:20.798381 8506 ts_tablet_manager.cc:1905] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.799151 8505 ts_tablet_manager.cc:1905] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.805117 8107 tablet_service.cc:1467] Processing CreateTablet for tablet 254a53d5d0774f95819a3adc636439e6 (DEFAULT_TABLE table=test-table [id=afcc54f247264c938c600bf7f0f76258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.805455 8107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 254a53d5d0774f95819a3adc636439e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.805502 8246 tablet_service.cc:1467] Processing CreateTablet for tablet 254a53d5d0774f95819a3adc636439e6 (DEFAULT_TABLE table=test-table [id=afcc54f247264c938c600bf7f0f76258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.805557 8385 tablet_service.cc:1467] Processing CreateTablet for tablet 254a53d5d0774f95819a3adc636439e6 (DEFAULT_TABLE table=test-table [id=afcc54f247264c938c600bf7f0f76258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:20.805867 8246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 254a53d5d0774f95819a3adc636439e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.805886 8385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 254a53d5d0774f95819a3adc636439e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:20.811231 8482 tablet_bootstrap.cc:492] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: Bootstrap starting.
I20260926 03:00:20.811951 8481 tablet_bootstrap.cc:492] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: Bootstrap starting.
I20260926 03:00:20.811959 8480 tablet_bootstrap.cc:492] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: Bootstrap starting.
I20260926 03:00:20.812805 8482 tablet_bootstrap.cc:654] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.813503 8481 tablet_bootstrap.cc:654] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.813529 8480 tablet_bootstrap.cc:654] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:20.816490 8506 ts_tablet_manager.cc:1918] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:20.817334 8506 log.cc:1196] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/fcdf75aa10ab40f597adb110d0e2a668
I20260926 03:00:20.817796 8505 ts_tablet_manager.cc:1918] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:20.817963 8505 log.cc:1196] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/fcdf75aa10ab40f597adb110d0e2a668
I20260926 03:00:20.818485 8505 ts_tablet_manager.cc:1939] T fcdf75aa10ab40f597adb110d0e2a668 P ae5a41317a5346bfa661ed4f148658b6: Deleting consensus metadata
I20260926 03:00:20.819087 8504 ts_tablet_manager.cc:1905] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:20.819648 8506 ts_tablet_manager.cc:1939] T fcdf75aa10ab40f597adb110d0e2a668 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting consensus metadata
I20260926 03:00:20.822026 7955 catalog_manager.cc:4841] TS f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775): tablet fcdf75aa10ab40f597adb110d0e2a668 (table test-table [id=3d7201e033b14c0891ab63d5d46f09d2]) successfully deleted
I20260926 03:00:20.823203 7956 catalog_manager.cc:4841] TS ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313): tablet fcdf75aa10ab40f597adb110d0e2a668 (table test-table [id=3d7201e033b14c0891ab63d5d46f09d2]) successfully deleted
W20260926 03:00:20.823697 7956 catalog_manager.cc:4808] TS ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313): delete failed for tablet fcdf75aa10ab40f597adb110d0e2a668 because the tablet was not found. No further retry: Not found: Tablet not found: fcdf75aa10ab40f597adb110d0e2a668
I20260926 03:00:20.824625 8482 tablet_bootstrap.cc:492] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: No bootstrap required, opened a new log
I20260926 03:00:20.824791 8482 ts_tablet_manager.cc:1397] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260926 03:00:20.825563 8482 raft_consensus.cc:348] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.825819 8482 raft_consensus.cc:374] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.825881 8482 raft_consensus.cc:729] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d99b9c9e454bd7b77f67f25bf45214, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.826084 8482 consensus_queue.cc:260] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.826534 8482 ts_tablet_manager.cc:1428] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:00:20.826785 8481 tablet_bootstrap.cc:492] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: No bootstrap required, opened a new log
I20260926 03:00:20.826973 8481 ts_tablet_manager.cc:1397] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260926 03:00:20.827646 8481 raft_consensus.cc:348] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.827905 8481 raft_consensus.cc:374] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.827982 8481 raft_consensus.cc:729] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962a1e07dcba40eb9d6e8d4a87b55d67, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.828262 8504 ts_tablet_manager.cc:1918] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:20.828233 8481 consensus_queue.cc:260] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.828711 8481 ts_tablet_manager.cc:1428] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:00:20.828428 8504 log.cc:1196] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/fcdf75aa10ab40f597adb110d0e2a668
I20260926 03:00:20.829579 8504 ts_tablet_manager.cc:1939] T fcdf75aa10ab40f597adb110d0e2a668 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting consensus metadata
I20260926 03:00:20.830406 7958 catalog_manager.cc:4841] TS 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077): tablet fcdf75aa10ab40f597adb110d0e2a668 (table test-table [id=3d7201e033b14c0891ab63d5d46f09d2]) successfully deleted
I20260926 03:00:20.832288 8480 tablet_bootstrap.cc:492] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: No bootstrap required, opened a new log
I20260926 03:00:20.832487 8480 ts_tablet_manager.cc:1397] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
I20260926 03:00:20.833297 8480 raft_consensus.cc:348] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.833535 8480 raft_consensus.cc:374] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:20.833622 8480 raft_consensus.cc:729] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5a41317a5346bfa661ed4f148658b6, State: Initialized, Role: FOLLOWER
I20260926 03:00:20.833913 8480 consensus_queue.cc:260] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.834499 8480 ts_tablet_manager.cc:1428] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:00:20.914793 8486 raft_consensus.cc:482] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:20.915060 8486 raft_consensus.cc:504] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.916846 8486 leader_election.cc:288] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775), ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313)
I20260926 03:00:20.926693 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.917639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43119 (local address 127.0.97.1:35775)
0926 03:00:20.917893 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.917901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.917926 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:00:20.918016 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.918024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.918116 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.918227 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:20.919150 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.920624 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.923704 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.925212 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.925327 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.925470 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.926123 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.926300 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 03:00:20.927676 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.917050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35775 (local address 127.0.97.2:43119)
0926 03:00:20.917552 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:20.917577 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:20.917710 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:20.918464 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:20.918472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:20.918517 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:20.918941 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.918956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.920933 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.920941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.923462 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.923483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.925637 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.925644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.925910 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.925960 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 03:00:20.926978 (+ 1018us) client_negotiation.cc:240] Negotiation successful
0926 03:00:20.927174 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":243,"threads_started":1}
I20260926 03:00:20.928381 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254a53d5d0774f95819a3adc636439e6" candidate_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" is_pre_election: true
I20260926 03:00:20.928740 8128 raft_consensus.cc:2454] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 962a1e07dcba40eb9d6e8d4a87b55d67 in term 0.
I20260926 03:00:20.929282 8198 leader_election.cc:302] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962a1e07dcba40eb9d6e8d4a87b55d67, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:20.929702 8486 raft_consensus.cc:2781] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:20.929836 8486 raft_consensus.cc:482] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:20.929947 8486 raft_consensus.cc:3037] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.932751 8486 raft_consensus.cc:504] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.933384 8486 leader_election.cc:288] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [CANDIDATE]: Term 1 election: Requested vote from peers f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775), ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313)
I20260926 03:00:20.933751 8128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254a53d5d0774f95819a3adc636439e6" candidate_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d99b9c9e454bd7b77f67f25bf45214"
I20260926 03:00:20.933969 8128 raft_consensus.cc:3037] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.936836 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.917161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38313 (local address 127.0.97.2:51215)
0926 03:00:20.929276 (+ 12115us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:20.929297 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:20.929409 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:20.930051 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:20.930056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:20.930079 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:20.930406 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.930417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.932113 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.932118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.934196 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:20.934210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.935988 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.935997 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:20.936243 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.936275 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:00:20.936357 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:00:20.936496 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11973,"thread_start_us":11608,"threads_started":1}
I20260926 03:00:20.936926 8128 raft_consensus.cc:2454] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962a1e07dcba40eb9d6e8d4a87b55d67 in term 1.
I20260926 03:00:20.937366 8198 leader_election.cc:302] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 962a1e07dcba40eb9d6e8d4a87b55d67, f0d99b9c9e454bd7b77f67f25bf45214; no voters:
I20260926 03:00:20.937727 8486 raft_consensus.cc:2781] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:20.937757 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.917991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51215 (local address 127.0.97.3:38313)
0926 03:00:20.929377 (+ 11386us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.929383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.929407 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:00:20.929679 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.929686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.929766 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.929865 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:20.930593 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.931967 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.934349 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.935601 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.935729 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.935844 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.937184 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.937352 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11221,"thread_start_us":11540,"threads_started":1}
I20260926 03:00:20.939018 8405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254a53d5d0774f95819a3adc636439e6" candidate_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5a41317a5346bfa661ed4f148658b6"
I20260926 03:00:20.939128 8406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254a53d5d0774f95819a3adc636439e6" candidate_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5a41317a5346bfa661ed4f148658b6" is_pre_election: true
I20260926 03:00:20.939280 8405 raft_consensus.cc:3037] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:20.940835 8486 raft_consensus.cc:686] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 LEADER]: Becoming Leader. State: Replica: 962a1e07dcba40eb9d6e8d4a87b55d67, State: Running, Role: LEADER
I20260926 03:00:20.941246 8486 consensus_queue.cc:237] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } }
I20260926 03:00:20.943836 8405 raft_consensus.cc:2454] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962a1e07dcba40eb9d6e8d4a87b55d67 in term 1.
I20260926 03:00:20.946476 7969 catalog_manager.cc:5495] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "962a1e07dcba40eb9d6e8d4a87b55d67" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 } health_report { overall_health: UNKNOWN } } }
I20260926 03:00:20.992621 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:20.985812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.0.97.2:39077)
0926 03:00:20.986967 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:20.986971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:20.986989 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:00:20.987061 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:20.987065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:20.987130 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:20.987303 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:20.987715 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.988716 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.990316 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:20.990766 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:20.990895 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:20.991428 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:20.992001 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:20.992155 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:20.992223 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 03:00:20.992294 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":838,"threads_started":1}
I20260926 03:00:21.007447 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.002086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58559 (local address 127.0.97.3:38313)
0926 03:00:21.002252 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:21.002256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:21.002731 (+ 475us) server_negotiation.cc:407] Connection header received
0926 03:00:21.002948 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:21.002953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:21.003031 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:21.003124 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:21.003633 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.004390 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.005789 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.006196 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.006328 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:21.006746 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:21.006846 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:21.006940 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:21.007079 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 03:00:21.007148 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:00:21.007558 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.002080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38313 (local address 127.0.97.2:58559)
0926 03:00:21.002630 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:21.002644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:21.002814 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:21.003148 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:21.003153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:21.003173 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:00:21.003471 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:21.003479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.004518 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.004525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:21.005651 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:21.005663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.006359 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.006366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:21.006518 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:21.006565 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:00:21.006940 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:00:21.006953 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 03:00:21.007046 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:00:21.007174 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":297,"threads_started":1}
I20260926 03:00:21.015422 8128 raft_consensus.cc:1261] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Refusing update from remote peer 962a1e07dcba40eb9d6e8d4a87b55d67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:21.015450 8405 raft_consensus.cc:1261] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Refusing update from remote peer 962a1e07dcba40eb9d6e8d4a87b55d67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:21.015987 8486 consensus_queue.cc:1035] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5a41317a5346bfa661ed4f148658b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:21.016320 8567 consensus_queue.cc:1035] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d99b9c9e454bd7b77f67f25bf45214" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:21.041777 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.035461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54913 (local address 127.0.97.2:39077)
0926 03:00:21.035715 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:21.035722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:21.036226 (+ 504us) server_negotiation.cc:407] Connection header received
0926 03:00:21.036509 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:21.036515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:21.036658 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:21.036774 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:21.037342 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.038482 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.039969 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.040386 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.040523 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:21.041027 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:21.041149 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:21.041209 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:21.041392 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 03:00:21.041461 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 03:00:21.041960 8576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.035410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39077 (local address 127.0.97.3:54913)
0926 03:00:21.036097 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:21.036114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:21.036326 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:21.036789 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:21.036793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:21.036808 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:00:21.037133 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:21.037171 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.038720 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.038723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:21.039825 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:21.039835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.040567 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.040576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:21.040768 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:21.040816 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:00:21.041271 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:00:21.041278 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:00:21.041362 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:00:21.041487 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":337,"threads_started":1}
I20260926 03:00:21.086021 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:21.086261 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:21.089892 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da1db226618346f59d834ea832eeb41d: Sending DeleteTablet for 3 replicas of tablet 254a53d5d0774f95819a3adc636439e6
I20260926 03:00:21.090725 8246 tablet_service.cc:1514] Processing DeleteTablet for tablet 254a53d5d0774f95819a3adc636439e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260926 03:00:21.091029 8504 tablet_replica.cc:330] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: stopping tablet replica
I20260926 03:00:21.091262 8504 raft_consensus.cc:2229] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:00:21.091226 8107 tablet_service.cc:1514] Processing DeleteTablet for tablet 254a53d5d0774f95819a3adc636439e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43094
I20260926 03:00:21.091502 8506 tablet_replica.cc:330] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: stopping tablet replica
I20260926 03:00:21.091676 8506 raft_consensus.cc:2229] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:21.091710 8504 raft_consensus.cc:2258] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:21.091840 8506 raft_consensus.cc:2258] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:21.093106 8506 ts_tablet_manager.cc:1905] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:21.097208 8385 tablet_service.cc:1514] Processing DeleteTablet for tablet 254a53d5d0774f95819a3adc636439e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:21 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39618
I20260926 03:00:21.097504 8505 tablet_replica.cc:330] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: stopping tablet replica
I20260926 03:00:21.097716 8505 raft_consensus.cc:2229] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:21.097934 8505 raft_consensus.cc:2258] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:21.099020 8505 ts_tablet_manager.cc:1905] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:21.102492 8504 ts_tablet_manager.cc:1905] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:21.107211 8505 ts_tablet_manager.cc:1918] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:00:21.107366 8505 log.cc:1196] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/254a53d5d0774f95819a3adc636439e6
I20260926 03:00:21.107775 8505 ts_tablet_manager.cc:1939] T 254a53d5d0774f95819a3adc636439e6 P ae5a41317a5346bfa661ed4f148658b6: Deleting consensus metadata
I20260926 03:00:21.108867 7956 catalog_manager.cc:4841] TS ae5a41317a5346bfa661ed4f148658b6 (127.0.97.3:38313): tablet 254a53d5d0774f95819a3adc636439e6 (table test-table [id=afcc54f247264c938c600bf7f0f76258]) successfully deleted
I20260926 03:00:21.110971 8506 ts_tablet_manager.cc:1918] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:00:21.111104 8506 log.cc:1196] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/254a53d5d0774f95819a3adc636439e6
I20260926 03:00:21.111474 8506 ts_tablet_manager.cc:1939] T 254a53d5d0774f95819a3adc636439e6 P f0d99b9c9e454bd7b77f67f25bf45214: Deleting consensus metadata
I20260926 03:00:21.112493 7955 catalog_manager.cc:4841] TS f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775): tablet 254a53d5d0774f95819a3adc636439e6 (table test-table [id=afcc54f247264c938c600bf7f0f76258]) successfully deleted
I20260926 03:00:21.112699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 8042
I20260926 03:00:21.123131 8504 ts_tablet_manager.cc:1918] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:00:21.125124 8504 log.cc:1196] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/254a53d5d0774f95819a3adc636439e6
I20260926 03:00:21.125667 8504 ts_tablet_manager.cc:1939] T 254a53d5d0774f95819a3adc636439e6 P 962a1e07dcba40eb9d6e8d4a87b55d67: Deleting consensus metadata
I20260926 03:00:21.130164 7958 catalog_manager.cc:4841] TS 962a1e07dcba40eb9d6e8d4a87b55d67 (127.0.97.2:39077): tablet 254a53d5d0774f95819a3adc636439e6 (table test-table [id=afcc54f247264c938c600bf7f0f76258]) successfully deleted
Sep 26 03:00:21 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260926 03:00:21.153184 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.113905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.0.97.1:35775)
0926 03:00:21.114076 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:21.114082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:21.133477 (+ 19395us) server_negotiation.cc:407] Connection header received
0926 03:00:21.133683 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:21.133688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:21.133771 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:21.133905 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:21.135596 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.136768 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.138714 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.139389 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.139484 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:21.144991 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:21.145111 (+ 120us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:21.145132 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:21.145183 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:21.149169 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:21.149997 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:21.150006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:21.150010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:21.150238 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:21.150687 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:21.150692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:21.150695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:21.151149 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:21.151456 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:21.152568 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0926 03:00:21.152779 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:00:21.154014 8173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:21.366604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 8042
I20260926 03:00:21.390712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 8182
Sep 26 03:00:21 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260926 03:00:21.406034 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.391491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.0.97.2:39077)
0926 03:00:21.391657 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:21.391663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:21.391703 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:00:21.391929 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:21.391935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:21.392031 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:21.392153 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:21.393378 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.394137 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.395327 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.395752 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.395884 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:21.400026 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:21.400038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:21.400053 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:21.400087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:21.403277 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:21.403919 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:21.403931 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:21.403933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:21.404069 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:21.404509 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:21.404515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:21.404520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:21.404878 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:21.405198 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:21.405487 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 03:00:21.405700 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:00:21.406761 8312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:21.486824 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.486004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35775 (local address 127.0.97.3:42967)
0926 03:00:21.486485 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:21.486624 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
W20260926 03:00:21.487167 8337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35775: connect: Connection refused (error 111)
I20260926 03:00:21.488539 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.488215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35775 (local address 127.0.97.3:35779)
0926 03:00:21.488357 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:21.488442 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 03:00:21.488821 8337 consensus_peers.cc:487] T 4f339db38634484dba2c4201aa508f5e P ae5a41317a5346bfa661ed4f148658b6 -> Peer f0d99b9c9e454bd7b77f67f25bf45214 (127.0.97.1:35775): Couldn't send request to peer f0d99b9c9e454bd7b77f67f25bf45214. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:00:21.537837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 8182
I20260926 03:00:21.559020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 8321
Sep 26 03:00:21 dist-test-slave-fccw krb5kdc[7926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260926 03:00:21.573742 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.559834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.0.97.3:38313)
0926 03:00:21.560288 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:21.560292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:21.560307 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:00:21.560358 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:21.560361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:21.560427 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:21.560515 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:21.561763 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.562406 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.563541 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:21.563936 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:21.564026 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:21.568127 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:21.568141 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:21.568166 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:21.568195 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:21.571399 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:21.571934 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:21.571940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:21.571942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:21.572062 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:21.572364 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:21.572367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:21.572369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:21.572664 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:21.572864 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:21.573312 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 03:00:21.573487 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":240,"threads_started":1}
I20260926 03:00:21.574409 8451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:21.716348 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:21.715872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39077 (local address 127.0.97.3:35527)
0926 03:00:21.716055 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:21.716156 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.2:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:00:21.716343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 8321
I20260926 03:00:21.741133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 7939
I20260926 03:00:21.742295 8011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:21.854986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 7939
2026-09-26T03:00:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5604 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:00:21 dist-test-slave-fccw krb5kdc[8590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:00:21 dist-test-slave-fccw krb5kdc[8590](info): set up 2 sockets
Sep 26 03:00:21 dist-test-slave-fccw krb5kdc[8590](info): commencing operation
krb5kdc: starting...
W20260926 03:00:22.937915 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:00:23 dist-test-slave-fccw krb5kdc[8590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:00:23.139557 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40207
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:23.450502 8603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:23.450769 8603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:23.450855 8603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:23.457949 8603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:23.458042 8603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:23.458151 8603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:23.458248 8603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:23.458325 8603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:23.467556 8603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:23.468765 8603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:23.470043 8603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:23.481925 8608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:23.481871 8611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:23.481843 8609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:23.482455 8603 server_base.cc:1034] running on GCE node
I20260926 03:00:23.483207 8603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:23.484537 8603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:23.485760 8603 hybrid_clock.cc:648] HybridClock initialized: now 1790391623485725 us; error 49 us; skew 500 ppm
I20260926 03:00:23.486078 8603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T03:00:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1699 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:00:23 dist-test-slave-fccw krb5kdc[8617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:00:23 dist-test-slave-fccw krb5kdc[8617](info): set up 2 sockets
Sep 26 03:00:23 dist-test-slave-fccw krb5kdc[8617](info): commencing operation
krb5kdc: starting...
W20260926 03:00:24.666285 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:00:24 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:00:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:00:24.884672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:25.199991 8630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:25.200311 8630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:25.200459 8630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:25.208073 8630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:25.208222 8630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:25.208314 8630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:25.208410 8630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:25.208473 8630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:25.218150 8630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:25.219398 8630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:25.220722 8630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:25.231822 8635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:25.231858 8638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:25.231896 8636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:25.232682 8630 server_base.cc:1034] running on GCE node
I20260926 03:00:25.233405 8630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:25.234642 8630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:25.235839 8630 hybrid_clock.cc:648] HybridClock initialized: now 1790391625235806 us; error 28 us; skew 500 ppm
I20260926 03:00:25.236110 8630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:25 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:25.243356 8630 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:00:25.244679 8630 webserver.cc:460] Webserver started at http://127.0.97.62:33691/ using document root <none> and password file <none>
I20260926 03:00:25.245353 8630 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:25.245457 8630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:25.245711 8630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:25.248500 8630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "82afe01df4a844d0b30e9cce930e5d2c"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "1f8c3533e0028eac36feb108ed80908b"
server_key_iv: "56e9100e6214d16cd5a3e4a90f3cf6fc"
server_key_version: "encryptionkey@0"
I20260926 03:00:25.249313 8630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "82afe01df4a844d0b30e9cce930e5d2c"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "1f8c3533e0028eac36feb108ed80908b"
server_key_iv: "56e9100e6214d16cd5a3e4a90f3cf6fc"
server_key_version: "encryptionkey@0"
I20260926 03:00:25.254257 8630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:00:25.257665 8645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:25.259068 8630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:00:25.259227 8630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "82afe01df4a844d0b30e9cce930e5d2c"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "1f8c3533e0028eac36feb108ed80908b"
server_key_iv: "56e9100e6214d16cd5a3e4a90f3cf6fc"
server_key_version: "encryptionkey@0"
I20260926 03:00:25.259413 8630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:25.276389 8630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:25.280105 8630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:25.280362 8630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:25.297827 8630 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43503
I20260926 03:00:25.297847 8696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43503 every 8 connection(s)
I20260926 03:00:25.299136 8630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:25.302538 8697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:25.303607 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 8630
I20260926 03:00:25.303830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:25.304633 388 external_mini_cluster.cc:1428] Setting key 35a61f19ca28a4861cd49b22c7aabaa1
I20260926 03:00:25.316167 8697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Bootstrap starting.
Sep 26 03:00:25 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:25.319614 8697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:25.320722 8697 log.cc:824] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:25.323422 8697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: No bootstrap required, opened a new log
I20260926 03:00:25.326439 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:25.306577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.0.97.62:43503)
0926 03:00:25.307324 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:25.307334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:25.307371 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:00:25.308890 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:25.308909 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:25.309337 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:25.309641 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:25.311415 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:25.312678 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:25.314465 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:25.315080 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:25.315202 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:25.319575 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:25.319582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:25.319594 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:25.319631 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:25.322861 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:25.323448 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:25.323455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:25.323462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:25.323667 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:25.324013 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:25.324017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:25.324020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:25.324482 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:25.324768 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:25.325193 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 03:00:25.325557 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":385,"threads_started":1}
I20260926 03:00:25.330708 8697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:25.331028 8697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:25.331127 8697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82afe01df4a844d0b30e9cce930e5d2c, State: Initialized, Role: FOLLOWER
I20260926 03:00:25.331713 8697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:25.331955 8697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:25.332087 8697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:25.332242 8697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:25.334509 8697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:25.334992 8697 leader_election.cc:302] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82afe01df4a844d0b30e9cce930e5d2c; no voters:
I20260926 03:00:25.335521 8697 leader_election.cc:288] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:25.335621 8702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:25.336385 8702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 LEADER]: Becoming Leader. State: Replica: 82afe01df4a844d0b30e9cce930e5d2c, State: Running, Role: LEADER
I20260926 03:00:25.336895 8702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:25.337625 8697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:25.339670 8703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82afe01df4a844d0b30e9cce930e5d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } } }
I20260926 03:00:25.339965 8703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:25.340670 8704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82afe01df4a844d0b30e9cce930e5d2c. Latest consensus state: current_term: 1 leader_uuid: "82afe01df4a844d0b30e9cce930e5d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } } }
I20260926 03:00:25.340945 8704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:25.347267 8707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:25.351202 8707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:25.358476 8707 catalog_manager.cc:1347] Generated new cluster ID: f13ce5aaf6e049759ce1a176c273ab84
I20260926 03:00:25.358580 8707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:25.377118 8707 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:25.378922 8707 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:25.391072 8707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Generated new TSK 0
I20260926 03:00:25.391979 8707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:00:25.450348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43503
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:25.773196 8723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:25.773520 8723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:25.773672 8723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:25.780570 8723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:25.780689 8723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:25.780874 8723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:25.790128 8723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:25.791256 8723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:25.792474 8723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:25.804878 8728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:25.804903 8731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:25.804925 8729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:25.805953 8723 server_base.cc:1034] running on GCE node
I20260926 03:00:25.806551 8723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:25.807397 8723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:25.808606 8723 hybrid_clock.cc:648] HybridClock initialized: now 1790391625808564 us; error 56 us; skew 500 ppm
I20260926 03:00:25.808889 8723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:25 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:25.811775 8723 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:00:25.813133 8723 webserver.cc:460] Webserver started at http://127.0.97.1:41693/ using document root <none> and password file <none>
I20260926 03:00:25.813719 8723 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:25.813812 8723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:25.814040 8723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:25.816607 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "749b8364af6c49379a86f3b0005d67d3"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "45c4cfdc9720bd0424c1966c978010eb"
server_key_iv: "4f20db21895c4edb602b58303011cc48"
server_key_version: "encryptionkey@0"
I20260926 03:00:25.817358 8723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "749b8364af6c49379a86f3b0005d67d3"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "45c4cfdc9720bd0424c1966c978010eb"
server_key_iv: "4f20db21895c4edb602b58303011cc48"
server_key_version: "encryptionkey@0"
I20260926 03:00:25.821969 8723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:00:25.825109 8738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:25.826467 8723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:00:25.826597 8723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "749b8364af6c49379a86f3b0005d67d3"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "45c4cfdc9720bd0424c1966c978010eb"
server_key_iv: "4f20db21895c4edb602b58303011cc48"
server_key_version: "encryptionkey@0"
I20260926 03:00:25.826743 8723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:25.839870 8723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:25.843547 8723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:25.843773 8723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:25.844578 8723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:25.845785 8723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:25.845851 8723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:25.845935 8723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:25.845980 8723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:25.877066 8723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37431
I20260926 03:00:25.877132 8850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37431 every 8 connection(s)
I20260926 03:00:25.878378 8723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:25.881063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 8723
I20260926 03:00:25.881269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:00:25.882068 388 external_mini_cluster.cc:1428] Setting key 6feee5f6bd0a972e0eebbc46bdaa3ac1
Sep 26 03:00:25 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:25.900611 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:25.880864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38581 (local address 127.0.97.62:43503)
0926 03:00:25.881382 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:25.881389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:25.882524 (+ 1135us) server_negotiation.cc:407] Connection header received
0926 03:00:25.884484 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:25.884488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:25.884547 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:25.884635 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:25.886740 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:25.887457 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:25.889885 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:25.890390 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:25.890518 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:25.894119 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:25.894128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:25.894135 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:25.894178 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:25.897477 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:25.898131 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:25.898139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:25.898146 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:25.898287 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:25.898686 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:25.898693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:25.898699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:25.899075 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:25.899225 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:25.899862 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 03:00:25.900112 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260926 03:00:25.901309 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:25.881453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:38581)
0926 03:00:25.882340 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:25.882392 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:25.884209 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:25.884901 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:25.884915 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:25.885731 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:25.886584 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:25.886612 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:25.887604 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:25.887613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:25.889756 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:25.889773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:25.890615 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:25.890625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:25.890958 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:25.891699 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:25.891720 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:25.893883 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:25.897606 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:25.897612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:25.897623 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:25.897998 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:25.898395 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:25.898398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:25.898401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:25.898569 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:25.899184 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:25.899189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:25.899486 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 03:00:25.899719 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 03:00:25.900086 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":320,"threads_started":1}
I20260926 03:00:25.902912 8851 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43503
I20260926 03:00:25.903242 8851 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:25.904137 8851 heartbeater.cc:511] Master 127.0.97.62:43503 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:00:25.906291 8662 ts_manager.cc:194] Registered new tserver with Master: 749b8364af6c49379a86f3b0005d67d3 (127.0.97.1:37431)
I20260926 03:00:25.907796 8662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:00:25.930280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43503
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:26.245095 8857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:26.245391 8857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:26.245546 8857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:26.252600 8857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:26.252727 8857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:26.252890 8857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:26.262370 8857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:26.263659 8857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:26.265022 8857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:26.277868 8862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:26.277837 8863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:26.278293 8865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:26.278782 8857 server_base.cc:1034] running on GCE node
I20260926 03:00:26.279307 8857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:26.280133 8857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:26.281314 8857 hybrid_clock.cc:648] HybridClock initialized: now 1790391626281263 us; error 41 us; skew 500 ppm
I20260926 03:00:26.281617 8857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:26 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:26.284618 8857 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:00:26.285989 8857 webserver.cc:460] Webserver started at http://127.0.97.2:44173/ using document root <none> and password file <none>
I20260926 03:00:26.286599 8857 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:26.286692 8857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:26.286932 8857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:26.289731 8857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "87c1162a802d4c80a413a90fc6fcaadd"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "5f0799e8f230ab984d7cb1c7c5418439"
server_key_iv: "432fa9e40f1e90fe2b34a6712c3d269b"
server_key_version: "encryptionkey@0"
I20260926 03:00:26.290400 8857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "87c1162a802d4c80a413a90fc6fcaadd"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "5f0799e8f230ab984d7cb1c7c5418439"
server_key_iv: "432fa9e40f1e90fe2b34a6712c3d269b"
server_key_version: "encryptionkey@0"
I20260926 03:00:26.294987 8857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:00:26.298276 8872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:26.299551 8857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:00:26.299705 8857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "87c1162a802d4c80a413a90fc6fcaadd"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "5f0799e8f230ab984d7cb1c7c5418439"
server_key_iv: "432fa9e40f1e90fe2b34a6712c3d269b"
server_key_version: "encryptionkey@0"
I20260926 03:00:26.299866 8857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:26.320281 8857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:26.324242 8857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:26.324458 8857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:26.325439 8857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:26.326645 8857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:26.326718 8857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:26.326879 8857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:26.326927 8857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:26.358258 8857 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34187
I20260926 03:00:26.358327 8984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34187 every 8 connection(s)
I20260926 03:00:26.359537 8857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:26.368537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 8857
I20260926 03:00:26.368728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:00:26.369210 388 external_mini_cluster.cc:1428] Setting key 752db3c2d81a81b267569bedef6bae13
Sep 26 03:00:26 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:26.379139 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:26.362078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54747 (local address 127.0.97.62:43503)
0926 03:00:26.362244 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:26.362248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:26.363499 (+ 1251us) server_negotiation.cc:407] Connection header received
0926 03:00:26.365519 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:26.365524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:26.365621 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:26.365729 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:26.367155 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.367861 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.369612 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.370151 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.370277 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:26.373766 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:26.373771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:26.373774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:26.373802 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:26.376251 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:26.376956 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:26.376960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:26.376962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:26.377143 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:26.377602 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:26.377605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:26.377607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:26.377846 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:26.377949 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:26.378753 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 03:00:26.378908 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:00:26.380636 8987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:26.362598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:54747)
0926 03:00:26.363352 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:26.363395 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:26.365265 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:26.365960 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:26.365968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:26.366453 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:26.366994 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:26.367010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.368016 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.368020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:26.369463 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:26.369474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.370373 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.370380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:26.370607 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:26.371328 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:26.371353 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:26.373575 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:26.376379 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:26.376389 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:26.376405 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:26.376803 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:26.377250 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:26.377256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:26.377261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:26.377492 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:26.377959 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:26.377969 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:26.378390 (+ 421us) client_negotiation.cc:769] Sending connection context
0926 03:00:26.378679 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 03:00:26.379212 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":314,"threads_started":1}
I20260926 03:00:26.381997 8985 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43503
I20260926 03:00:26.382277 8985 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:26.382964 8985 heartbeater.cc:511] Master 127.0.97.62:43503 requested a full tablet report, sending...
I20260926 03:00:26.384320 8662 ts_manager.cc:194] Registered new tserver with Master: 87c1162a802d4c80a413a90fc6fcaadd (127.0.97.2:34187)
I20260926 03:00:26.385135 8662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54747
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:00:26.415799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43503
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:26.727103 8990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:26.727430 8990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:26.727593 8990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:26.734763 8990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:26.734912 8990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:26.735086 8990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:26.744729 8990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:26.746047 8990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:26.747355 8990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:26.759982 8996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:26.759963 8998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:26.760197 8995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:26.760628 8990 server_base.cc:1034] running on GCE node
I20260926 03:00:26.761286 8990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:26.762253 8990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:26.763463 8990 hybrid_clock.cc:648] HybridClock initialized: now 1790391626763406 us; error 49 us; skew 500 ppm
I20260926 03:00:26.763767 8990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:26 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:26.766721 8990 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:00:26.768020 8990 webserver.cc:460] Webserver started at http://127.0.97.3:37431/ using document root <none> and password file <none>
I20260926 03:00:26.768661 8990 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:26.768786 8990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:26.769131 8990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:26.771849 8990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "d59c7438614b46909b7e316ad2767c4e"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "91f0ab4a433f757a1c43667c0ba0c8ef"
server_key_iv: "11a1ab2fef99b3729327d4a85c1297e8"
server_key_version: "encryptionkey@0"
I20260926 03:00:26.772598 8990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d59c7438614b46909b7e316ad2767c4e"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "91f0ab4a433f757a1c43667c0ba0c8ef"
server_key_iv: "11a1ab2fef99b3729327d4a85c1297e8"
server_key_version: "encryptionkey@0"
I20260926 03:00:26.777449 8990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260926 03:00:26.780570 9005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:26.781873 8990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:26.782023 8990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "d59c7438614b46909b7e316ad2767c4e"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "91f0ab4a433f757a1c43667c0ba0c8ef"
server_key_iv: "11a1ab2fef99b3729327d4a85c1297e8"
server_key_version: "encryptionkey@0"
I20260926 03:00:26.782210 8990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:26.805420 8990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:26.809110 8990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:26.809399 8990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:26.810304 8990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:26.811517 8990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:26.811592 8990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:26.811745 8990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:26.811794 8990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:26.843961 8990 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37989
I20260926 03:00:26.844051 9117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37989 every 8 connection(s)
I20260926 03:00:26.845256 8990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:26.854271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 8990
I20260926 03:00:26.854429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:00:26.854846 388 external_mini_cluster.cc:1428] Setting key bbda816069155f5036694c56218ae2c5
Sep 26 03:00:26 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:26.864331 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:26.847652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48717 (local address 127.0.97.62:43503)
0926 03:00:26.847815 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:26.847819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:26.848947 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 03:00:26.850714 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:26.850718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:26.850781 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:26.850867 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:26.852219 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.852943 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.854610 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.855056 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.855178 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:26.858882 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:26.858887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:26.858890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:26.858922 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:26.861356 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:26.862052 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:26.862059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:26.862062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:26.862174 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:26.862624 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:26.862633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:26.862636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:26.862893 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:26.863038 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:26.863787 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 03:00:26.863986 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:00:26.865346 9120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:26.848142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:48717)
0926 03:00:26.848796 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:26.848831 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:26.850478 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:26.851066 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:26.851074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:26.851539 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:26.852083 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:26.852099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.853148 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.853152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:26.854453 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:26.854465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:26.855280 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:26.855287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:26.855546 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:26.856335 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:26.856366 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:26.858685 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:26.861485 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:26.861493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:26.861511 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:26.861912 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:26.862279 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:26.862285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:26.862288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:26.862500 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:26.863031 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:26.863038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:26.863428 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 03:00:26.863703 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 03:00:26.864206 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":255,"threads_started":1}
I20260926 03:00:26.866678 9118 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43503
I20260926 03:00:26.866923 9118 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:26.867549 9118 heartbeater.cc:511] Master 127.0.97.62:43503 requested a full tablet report, sending...
I20260926 03:00:26.868884 8662 ts_manager.cc:194] Registered new tserver with Master: d59c7438614b46909b7e316ad2767c4e (127.0.97.3:37989)
I20260926 03:00:26.869815 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:00:26.869807 8662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48717
W20260926 03:00:26.880425 8847 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.1:37431
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
I20260926 03:00:26.910626 8851 heartbeater.cc:503] Master 127.0.97.62:43503 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260926 03:00:26.996717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 8723
Sep 26 03:00:27 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:00:27.022835 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:26.998096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36710 (local address 127.0.97.1:37431)
0926 03:00:26.999596 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:26.999606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:26.999645 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:00:26.999725 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:26.999736 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:27.000054 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:27.000235 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:27.001967 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:27.003702 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:27.004920 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:27.005960 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:27.006122 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:27.015819 (+ 9697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:27.015835 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:27.015856 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:27.015907 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:27.019833 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:27.020486 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:27.020497 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:27.020500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:27.020668 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:27.021089 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:27.021097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:27.021101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:27.021569 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:27.021874 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:27.022049 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 03:00:27.022329 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1269,"thread_start_us":1074,"threads_started":1}
I20260926 03:00:27.024181 8846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:27.110548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 8723
I20260926 03:00:27.127025 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 8857
Sep 26 03:00:27 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:00:27.141958 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:27.127811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.0.97.2:34187)
0926 03:00:27.128328 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:27.128335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:27.128359 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:00:27.128433 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:27.128439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:27.128640 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:27.128808 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:27.129966 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:27.130909 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:27.132076 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:27.132599 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:27.132727 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:27.136598 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:27.136608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:27.136621 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:27.136651 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:27.139323 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:27.139970 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:27.139976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:27.139978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:27.140094 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:27.140435 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:27.140442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:27.140445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:27.140766 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:27.141082 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:27.141397 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:00:27.141643 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":248,"threads_started":1}
I20260926 03:00:27.143229 8980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:27.223443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 8857
I20260926 03:00:27.239224 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 8990
Sep 26 03:00:27 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:00:27.254577 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:27.240264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.0.97.3:37989)
0926 03:00:27.240878 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:27.240884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:27.240913 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:00:27.240969 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:27.240975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:27.241259 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:27.241379 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:27.242568 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:27.243695 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:27.244920 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:27.245603 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:27.245704 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:27.249749 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:27.249758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:27.249771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:27.249800 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:27.252196 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:27.252727 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:27.252732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:27.252734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:27.252847 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:27.253214 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:27.253217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:27.253219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:27.253517 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:27.253746 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:27.253914 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:00:27.254089 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":307,"threads_started":1}
I20260926 03:00:27.255810 9113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:27.343196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 8990
I20260926 03:00:27.358983 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 8630
I20260926 03:00:27.360067 8692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:27.460778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 8630
I20260926 03:00:27.481209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43503
--webserver_interface=127.0.97.62
--webserver_port=33691
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:27.793404 9135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:27.793718 9135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:27.793871 9135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:27.800724 9135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:27.800858 9135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:27.800957 9135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:27.801074 9135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:27.801141 9135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:27.810328 9135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:27.811607 9135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:27.812897 9135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:27.824158 9141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:27.824235 9143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:27.824115 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:27.825173 9135 server_base.cc:1034] running on GCE node
I20260926 03:00:27.825949 9135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:27.827149 9135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:27.828342 9135 hybrid_clock.cc:648] HybridClock initialized: now 1790391627828303 us; error 29 us; skew 500 ppm
I20260926 03:00:27.828701 9135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:27 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:27.831574 9135 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:00:27.832823 9135 webserver.cc:460] Webserver started at http://127.0.97.62:33691/ using document root <none> and password file <none>
I20260926 03:00:27.833467 9135 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:27.833570 9135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:27.838406 9135 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:00:27.840955 9150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:27.842247 9135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:27.842418 9135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "82afe01df4a844d0b30e9cce930e5d2c"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "1f8c3533e0028eac36feb108ed80908b"
server_key_iv: "56e9100e6214d16cd5a3e4a90f3cf6fc"
server_key_version: "encryptionkey@0"
I20260926 03:00:27.843308 9135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:27.877189 9135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:27.881074 9135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:27.881352 9135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:27.898706 9135 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43503
I20260926 03:00:27.898741 9201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43503 every 8 connection(s)
I20260926 03:00:27.900157 9135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:27.906587 9202 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:00:27.909628 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Bootstrap starting.
I20260926 03:00:27.910166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 9135
I20260926 03:00:27.910883 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37431
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43503
--webserver_port=41693
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:00:27.927484 9202 log.cc:824] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:27.935820 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:00:27.936333 9202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Bootstrap complete.
I20260926 03:00:27.945174 9202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:27.945968 9202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82afe01df4a844d0b30e9cce930e5d2c, State: Initialized, Role: FOLLOWER
I20260926 03:00:27.946612 9202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:27.946835 9202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:27.946978 9202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:27.947162 9202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:00:27.950829 9202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:27.951278 9202 leader_election.cc:302] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82afe01df4a844d0b30e9cce930e5d2c; no voters:
I20260926 03:00:27.951918 9202 leader_election.cc:288] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:00:27.952054 9206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:00:27.953174 9206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [term 2 LEADER]: Becoming Leader. State: Replica: 82afe01df4a844d0b30e9cce930e5d2c, State: Running, Role: LEADER
I20260926 03:00:27.953756 9206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } }
I20260926 03:00:27.954089 9202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:27.956346 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82afe01df4a844d0b30e9cce930e5d2c. Latest consensus state: current_term: 2 leader_uuid: "82afe01df4a844d0b30e9cce930e5d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } } }
I20260926 03:00:27.956813 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:27.957401 9207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "82afe01df4a844d0b30e9cce930e5d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82afe01df4a844d0b30e9cce930e5d2c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43503 } } }
I20260926 03:00:27.957733 9207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:27.964780 9212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:27.970789 9212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:27.972141 9212 catalog_manager.cc:1259] Loaded cluster ID: f13ce5aaf6e049759ce1a176c273ab84
I20260926 03:00:27.972286 9212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:27.976102 9212 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:27.977486 9212 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 82afe01df4a844d0b30e9cce930e5d2c: Loaded TSK: 0
I20260926 03:00:27.978976 9212 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:00:28.237927 9204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:28.238240 9204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:28.238397 9204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:28.245682 9204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:28.245818 9204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:28.245990 9204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:28.255864 9204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41693
--tserver_master_addrs=127.0.97.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:28.257246 9204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:28.258562 9204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:28.272058 9229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:28.272346 9228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:28.272212 9231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:28.281477 9204 server_base.cc:1034] running on GCE node
I20260926 03:00:28.282179 9204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:28.283191 9204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:28.284411 9204 hybrid_clock.cc:648] HybridClock initialized: now 1790391628284354 us; error 58 us; skew 500 ppm
I20260926 03:00:28.284726 9204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391628, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:28.287848 9204 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260926 03:00:28.289448 9204 webserver.cc:460] Webserver started at http://127.0.97.1:41693/ using document root <none> and password file <none>
I20260926 03:00:28.290234 9204 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:28.290371 9204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:28.295538 9204 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:00:28.298172 9238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:28.299518 9204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:00:28.299681 9204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "749b8364af6c49379a86f3b0005d67d3"
format_stamp: "Formatted at 2026-09-26 03:00:25 on dist-test-slave-fccw"
server_key: "45c4cfdc9720bd0424c1966c978010eb"
server_key_iv: "4f20db21895c4edb602b58303011cc48"
server_key_version: "encryptionkey@0"
I20260926 03:00:28.300635 9204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:28.315734 9204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:28.319459 9204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:28.319715 9204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:28.320639 9204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:28.321923 9204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:28.322027 9204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:28.322151 9204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:28.322249 9204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:28.354250 9204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37431
I20260926 03:00:28.354350 9350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37431 every 8 connection(s)
I20260926 03:00:28.355729 9204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:28.361685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 9204
I20260926 03:00:28.362300 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34187
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43503
--webserver_port=44173
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:28.373930 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:28.358788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:60865)
0926 03:00:28.359557 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:28.359598 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:28.361361 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:28.362872 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:28.362883 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:28.363455 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:28.363983 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.364000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.365637 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.365642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.367073 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.367085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.368083 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.368090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.368361 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.369057 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:28.369080 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:28.372614 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":359,"threads_started":1}
W20260926 03:00:28.373958 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:28.358706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60865 (local address 127.0.97.62:43503)
0926 03:00:28.359460 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:28.359475 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:28.359709 (+ 234us) server_negotiation.cc:407] Connection header received
0926 03:00:28.361934 (+ 2225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:28.361947 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:28.362300 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:28.362659 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:28.364184 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.365474 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.367211 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.367862 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.368053 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.372956 (+ 4903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60865: BlockingRecv error: recv got EOF from 127.0.97.1:60865 (error 108)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":255,"threads_started":1}
W20260926 03:00:28.375250 9351 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43503 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43503: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:28.384670 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:28.375823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:47081)
0926 03:00:28.376023 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:28.376044 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:28.376150 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:28.376434 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:28.376442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:28.376871 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:28.377123 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.377139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.378115 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.378123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.379847 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.379864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.380479 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.380488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.380709 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.381331 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:28.381360 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:28.384431 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:00:28.385151 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:28.376006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47081 (local address 127.0.97.62:43503)
0926 03:00:28.376143 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:28.376147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:28.376162 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:00:28.376252 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:28.376256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:28.376306 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:28.376380 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:28.377260 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.377978 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.379982 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.380339 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.380424 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.384671 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47081: BlockingRecv error: recv got EOF from 127.0.97.1:47081 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:28.393651 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:28.385313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:51841)
0926 03:00:28.385480 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:28.385498 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:28.385586 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:28.385887 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:28.385892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:28.386249 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:28.386413 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.386423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.387339 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.387345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.388775 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.388788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.389482 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.389488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.389657 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.390193 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:28.390213 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:28.393372 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:00:28.393968 9351 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:00:28.394110 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:28.385461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51841 (local address 127.0.97.62:43503)
0926 03:00:28.385616 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:28.385621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:28.385640 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:00:28.385689 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:28.385694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:28.385773 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:28.385847 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:28.386533 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.387214 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.388894 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.389356 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.389449 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.393535 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51841: BlockingRecv error: recv got EOF from 127.0.97.1:51841 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 03:00:28.682674 9355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:28.682907 9355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:28.683070 9355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:28.690032 9355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:28.690126 9355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:28.690295 9355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:28.699514 9355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44173
--tserver_master_addrs=127.0.97.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:28.700820 9355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:28.702183 9355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:28.715773 9361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:28.715831 9360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:28.715780 9363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:28.717051 9355 server_base.cc:1034] running on GCE node
I20260926 03:00:28.717682 9355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:28.718523 9355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:28.719746 9355 hybrid_clock.cc:648] HybridClock initialized: now 1790391628719709 us; error 46 us; skew 500 ppm
I20260926 03:00:28.720042 9355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391628, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:28.722868 9355 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260926 03:00:28.724128 9355 webserver.cc:460] Webserver started at http://127.0.97.2:44173/ using document root <none> and password file <none>
I20260926 03:00:28.724787 9355 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:28.724952 9355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:28.729925 9355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:00:28.732494 9370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:28.733776 9355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:00:28.733950 9355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "87c1162a802d4c80a413a90fc6fcaadd"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "5f0799e8f230ab984d7cb1c7c5418439"
server_key_iv: "432fa9e40f1e90fe2b34a6712c3d269b"
server_key_version: "encryptionkey@0"
I20260926 03:00:28.734858 9355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:28.760740 9355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:28.764518 9355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:28.764779 9355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:28.765808 9355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:28.767129 9355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:28.767246 9355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:28.767374 9355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:28.767457 9355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:28.799008 9355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34187
I20260926 03:00:28.799090 9482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34187 every 8 connection(s)
I20260926 03:00:28.800493 9355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:28.804567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 9355
I20260926 03:00:28.805347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37989
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43503
--webserver_port=37431
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:00:28.818462 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:28.802965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42215 (local address 127.0.97.62:43503)
0926 03:00:28.803179 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:28.803183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:28.804443 (+ 1260us) server_negotiation.cc:407] Connection header received
0926 03:00:28.807370 (+ 2927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:28.807374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:28.807439 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:28.807525 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:28.809499 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.810158 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.812587 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.813315 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.813463 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.818218 (+ 4755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42215: BlockingRecv error: recv got EOF from 127.0.97.2:42215 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 03:00:28.819141 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:28.803468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:42215)
0926 03:00:28.804259 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:28.804316 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:28.807079 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:28.807784 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:28.807797 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:28.808537 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:28.809334 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.809360 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.810310 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.810318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.812423 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.812440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.813528 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.813533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.813727 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.814480 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:28.814502 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:28.818062 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":269,"threads_started":1}
W20260926 03:00:28.819761 9483 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43503 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43503: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:28.830477 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:28.820326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:51719)
0926 03:00:28.820571 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:28.820591 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:28.820699 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:28.820979 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:28.820986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:28.821440 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:28.821632 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.821645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.822627 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.822635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.824463 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.824497 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.825516 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.825525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.825800 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.826408 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:28.826454 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:28.830184 (+ 3730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130}
W20260926 03:00:28.830662 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:28.820429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51719 (local address 127.0.97.62:43503)
0926 03:00:28.820644 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:28.820660 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:28.820674 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:00:28.820800 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:28.820803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:28.820853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:28.820926 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:28.821772 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.822478 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.824646 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.825320 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.825436 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.830421 (+ 4985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51719: BlockingRecv error: recv got EOF from 127.0.97.2:51719 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:28 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:28.840512 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:28.831559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:51579)
0926 03:00:28.831772 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:28.831789 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:28.831874 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:28.832234 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:28.832240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:28.832591 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:28.832756 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.832766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.834153 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.834159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.835589 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:28.835602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.836173 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.836178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:28.836360 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.836859 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:28.836879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:28.840239 (+ 3360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260926 03:00:28.840545 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:28.831687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51579 (local address 127.0.97.62:43503)
0926 03:00:28.831866 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:28.831871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:28.831888 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:00:28.832009 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:28.832014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:28.832089 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:28.832181 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:28.832897 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.834011 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.835708 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:28.836052 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:28.836136 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:28.840390 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51579: BlockingRecv error: recv got EOF from 127.0.97.2:51579 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 03:00:28.840982 9483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:00:29.133639 9486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:29.133882 9486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:29.134047 9486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:29.141171 9486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:29.141281 9486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:29.141453 9486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:29.150802 9486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37431
--tserver_master_addrs=127.0.97.62:43503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:29.152026 9486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:29.153275 9486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:29.165632 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:29.165675 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:29.165709 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:29.166419 9486 server_base.cc:1034] running on GCE node
I20260926 03:00:29.166950 9486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:29.167811 9486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:29.169057 9486 hybrid_clock.cc:648] HybridClock initialized: now 1790391629169007 us; error 39 us; skew 500 ppm
I20260926 03:00:29.169353 9486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391629, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:29.172322 9486 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260926 03:00:29.173748 9486 webserver.cc:460] Webserver started at http://127.0.97.3:37431/ using document root <none> and password file <none>
I20260926 03:00:29.174397 9486 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:29.174541 9486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:29.179380 9486 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:00:29.181805 9501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:29.182960 9486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:00:29.183120 9486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "d59c7438614b46909b7e316ad2767c4e"
format_stamp: "Formatted at 2026-09-26 03:00:26 on dist-test-slave-fccw"
server_key: "91f0ab4a433f757a1c43667c0ba0c8ef"
server_key_iv: "11a1ab2fef99b3729327d4a85c1297e8"
server_key_version: "encryptionkey@0"
I20260926 03:00:29.183980 9486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:29.206980 9486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:29.210549 9486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:29.210832 9486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:29.211759 9486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:29.212983 9486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:29.213138 9486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:29.213281 9486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:29.213383 9486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:29.244614 9486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37989
I20260926 03:00:29.244693 9613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37989 every 8 connection(s)
I20260926 03:00:29.246044 9486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:29.248733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 9486
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:00:29.261829 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:29.248563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37395 (local address 127.0.97.62:43503)
0926 03:00:29.248771 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:29.248777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:29.250211 (+ 1434us) server_negotiation.cc:407] Connection header received
0926 03:00:29.252326 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:29.252331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:29.252427 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:29.252531 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:29.253964 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.254713 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.256247 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.256919 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.257084 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.261584 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37395: BlockingRecv error: recv got EOF from 127.0.97.3:37395 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 03:00:29.262408 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:29.249163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:37395)
0926 03:00:29.250018 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:29.250075 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:29.252091 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:29.252741 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:29.252749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:29.253293 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:29.253800 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.253816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.254842 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.254850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.256108 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.256124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.257151 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.257155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.257331 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.257907 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:29.257928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:29.261417 (+ 3489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":326,"threads_started":1}
W20260926 03:00:29.263101 9614 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43503 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:43503: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260926 03:00:29.265081 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:29.249730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.0.97.62:43503)
0926 03:00:29.250765 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:29.250769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:29.251242 (+ 473us) server_negotiation.cc:407] Connection header received
0926 03:00:29.251435 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:29.251440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:29.251516 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:29.251629 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:29.253158 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.253865 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.255108 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.255545 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.255666 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.258482 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:29.258490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:29.258503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:29.258538 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:29.261539 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:29.262220 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:29.262228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:29.262236 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:29.262383 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:29.262740 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:29.262746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:29.262749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:29.263320 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:29.263597 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:29.264237 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 03:00:29.264579 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":853,"threads_started":1}
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:29.271162 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:29.263655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:45871)
0926 03:00:29.263819 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:29.263835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:29.263921 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:29.264296 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:29.264300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:29.264657 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:29.264843 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.264852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.266077 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.266084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.267151 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.267165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.267887 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.267892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.268022 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.268532 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:29.268548 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:29.270902 (+ 2354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:00:29.271256 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:29.263731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45871 (local address 127.0.97.62:43503)
0926 03:00:29.263877 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:29.263883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:29.263898 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:00:29.264006 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:29.264014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:29.264076 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:29.264155 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:29.264979 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.265948 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.267289 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.267761 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.267874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.271055 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45871: BlockingRecv error: recv got EOF from 127.0.97.3:45871 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:29.278134 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:29.271881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:59021)
0926 03:00:29.272047 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:29.272066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:29.272152 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:29.272362 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:29.272365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:29.272600 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:29.272721 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.272728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.273660 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.273663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.274504 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.274513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.275054 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.275058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.275173 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.275649 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:29.275665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:29.277949 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:00:29.278309 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:29.271952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59021 (local address 127.0.97.62:43503)
0926 03:00:29.272137 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:29.272141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:29.272154 (+ 13us) server_negotiation.cc:407] Connection header received
0926 03:00:29.272220 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:29.272223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:29.272265 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:29.272333 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:29.272822 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.273535 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.274608 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.274955 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.275036 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.278091 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59021: BlockingRecv error: recv got EOF from 127.0.97.3:59021 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260926 03:00:29.278381 9614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:00:29.357870 9347 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.1:37431
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:29.402431 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:29.394535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:37869)
0926 03:00:29.395042 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:29.395059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:29.395169 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:29.395481 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:29.395486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:29.395996 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:29.396233 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.396261 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.397490 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.397494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.398408 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.398418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.399054 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.399058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.399183 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.399721 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:29.399737 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:29.402221 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":278,"threads_started":1}
W20260926 03:00:29.402585 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:29.394657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37869 (local address 127.0.97.62:43503)
0926 03:00:29.394806 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:29.394810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:29.395127 (+ 317us) server_negotiation.cc:407] Connection header received
0926 03:00:29.395260 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:29.395263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:29.395336 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:29.395417 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:29.396392 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.397368 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.398526 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.398917 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.399009 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.402390 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37869: BlockingRecv error: recv got EOF from 127.0.97.1:37869 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:29 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:29.849390 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:29.841502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:51839)
0926 03:00:29.841945 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:29.841961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:29.842069 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:29.842325 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:29.842328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:29.842761 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:29.842945 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.842953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.843905 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.843908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.844816 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:29.844826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.845486 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.845490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:29.845602 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.846149 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:29.846165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:29.849059 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":240,"threads_started":1}
W20260926 03:00:29.849495 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:29.841543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51839 (local address 127.0.97.62:43503)
0926 03:00:29.841785 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:29.841792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:29.842035 (+ 243us) server_negotiation.cc:407] Connection header received
0926 03:00:29.842156 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:29.842159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:29.842215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:29.842288 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:29.843058 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.843780 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.844934 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:29.845340 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:29.845436 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:29.849222 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51839: BlockingRecv error: recv got EOF from 127.0.97.2:51839 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:00:30 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:30 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:30.292388 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:30.278898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:39471)
0926 03:00:30.279373 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:30.279406 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:30.279518 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:30.279789 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:30.279792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:30.280206 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:30.280386 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:30.280393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.281355 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.281359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:30.282257 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:30.282267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.282873 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.282878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:30.282991 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:30.289301 (+ 6310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:30.289318 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:30.292183 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":256,"threads_started":1}
W20260926 03:00:30.292546 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:30.278967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39471 (local address 127.0.97.62:43503)
0926 03:00:30.279146 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:30.279153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:30.279482 (+ 329us) server_negotiation.cc:407] Connection header received
0926 03:00:30.279616 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:30.279619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:30.279675 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:30.279752 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:30.280493 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.281246 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.282362 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.282738 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.282823 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:30.292347 (+ 9524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39471: BlockingRecv error: recv got EOF from 127.0.97.3:39471 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 03:00:30 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:30 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:30.411213 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:30.403290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:39375)
0926 03:00:30.403949 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:30.403971 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:30.404112 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:30.404438 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:30.404463 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:30.404850 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:30.405062 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:30.405077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.406371 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.406375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:30.407255 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:30.407266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.407875 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.407879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:30.407983 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:30.408504 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:30.408520 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:30.410989 (+ 2469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":361,"threads_started":1}
W20260926 03:00:30.411361 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:30.403374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39375 (local address 127.0.97.62:43503)
0926 03:00:30.403693 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:30.403702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:30.404047 (+ 345us) server_negotiation.cc:407] Connection header received
0926 03:00:30.404256 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:30.404259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:30.404315 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:30.404411 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:30.405211 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.406253 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.407370 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.407757 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.407850 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:30.411150 (+ 3300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39375: BlockingRecv error: recv got EOF from 127.0.97.1:39375 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Sep 26 03:00:30 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:30 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:30.858167 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:30.850263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:58823)
0926 03:00:30.850756 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:30.850771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:30.850920 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:30.851221 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:30.851226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:30.851665 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:30.851809 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:30.851817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.852819 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.852825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:30.853809 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:30.853819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.854507 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.854513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:30.854686 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:30.855292 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:30.855308 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:30.857826 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":276,"threads_started":1}
W20260926 03:00:30.858227 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:30.850323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58823 (local address 127.0.97.62:43503)
0926 03:00:30.850593 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:30.850601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:30.850849 (+ 248us) server_negotiation.cc:407] Connection header received
0926 03:00:30.851041 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:30.851050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:30.851105 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:30.851197 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:30.851941 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.852694 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.853938 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:30.854366 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:30.854498 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:30.858016 (+ 3518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58823: BlockingRecv error: recv got EOF from 127.0.97.2:58823 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 03:00:31 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:31 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:31.300843 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:31.293172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:52193)
0926 03:00:31.293691 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:31.293707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:31.293815 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:31.294089 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:31.294093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:31.294473 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:31.294620 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:31.294629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.295564 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.295568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:31.296436 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:31.296446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.297099 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.297103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:31.297209 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:31.297733 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:31.297748 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:31.300515 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
W20260926 03:00:31.301051 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:31.293335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52193 (local address 127.0.97.62:43503)
0926 03:00:31.293493 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:31.293497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:31.293776 (+ 279us) server_negotiation.cc:407] Connection header received
0926 03:00:31.293908 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:31.293912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:31.293976 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:31.294055 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:31.294735 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.295437 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.296544 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.296968 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.297070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:31.300727 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52193: BlockingRecv error: recv got EOF from 127.0.97.3:52193 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 03:00:31 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:31 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:31.420660 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:31.412210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:59597)
0926 03:00:31.412876 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:31.412894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:31.413097 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:31.413433 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:31.413439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:31.413799 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:31.413972 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:31.413981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.415010 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.415015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:31.415892 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:31.415902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.416588 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.416593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:31.416769 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:31.417414 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:31.417431 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:31.420412 (+ 2981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":368,"threads_started":1}
W20260926 03:00:31.420801 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:31.412396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59597 (local address 127.0.97.62:43503)
0926 03:00:31.412667 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:31.412675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:31.413026 (+ 351us) server_negotiation.cc:407] Connection header received
0926 03:00:31.413224 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:31.413230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:31.413297 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:31.413399 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:31.414119 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.414857 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.416039 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.416437 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.416576 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:31.420601 (+ 4025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59597: BlockingRecv error: recv got EOF from 127.0.97.1:59597 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 03:00:31 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:31 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:31.867133 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:31.859193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:48829)
0926 03:00:31.859725 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:31.859741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:31.859846 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:31.860158 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:31.860161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:31.860472 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:31.860630 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:31.860637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.861742 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.861749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:31.862607 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:31.862617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.863293 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.863299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:31.863477 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:31.864027 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:31.864043 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:31.866888 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":296,"threads_started":1}
W20260926 03:00:31.867288 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:31.859219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48829 (local address 127.0.97.62:43503)
0926 03:00:31.859430 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:31.859435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:31.859816 (+ 381us) server_negotiation.cc:407] Connection header received
0926 03:00:31.859973 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:31.859977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:31.860038 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:31.860124 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:31.860773 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.861612 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.862752 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:31.863157 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:31.863297 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:31.867065 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48829: BlockingRecv error: recv got EOF from 127.0.97.2:48829 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:00:32 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:32 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:32.309856 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:32.302019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:45105)
0926 03:00:32.302504 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:32.302520 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:32.302626 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:32.302929 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:32.302933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:32.303249 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:32.303396 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:32.303404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.304443 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.304449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:32.305837 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:32.305848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.306426 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.306430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:32.306535 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:32.307101 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:32.307117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:32.309623 (+ 2506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":277,"threads_started":1}
W20260926 03:00:32.309993 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:32.302113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45105 (local address 127.0.97.62:43503)
0926 03:00:32.302262 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:32.302266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:32.302586 (+ 320us) server_negotiation.cc:407] Connection header received
0926 03:00:32.302748 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:32.302752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:32.302820 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:32.302894 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:32.303508 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.304302 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.305946 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.306313 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.306398 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:32.309801 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45105: BlockingRecv error: recv got EOF from 127.0.97.3:45105 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 03:00:32 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:32 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:32.429651 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:32.421703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:37083)
0926 03:00:32.422231 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:32.422247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:32.422397 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:32.422717 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:32.422721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:32.423126 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:32.423261 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:32.423269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.424204 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.424211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:32.425236 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:32.425247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.425905 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.425911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:32.426105 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:32.426632 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:32.426648 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:32.429437 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
W20260926 03:00:32.429867 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:32.421755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37083 (local address 127.0.97.62:43503)
0926 03:00:32.421980 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:32.421988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:32.422327 (+ 339us) server_negotiation.cc:407] Connection header received
0926 03:00:32.422517 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:32.422532 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:32.422590 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:32.422690 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:32.423390 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.424086 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.425365 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.425765 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.425904 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:32.429612 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37083: BlockingRecv error: recv got EOF from 127.0.97.1:37083 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:00:32 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:32 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:32.875557 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:32.867941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:41297)
0926 03:00:32.868381 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:32.868397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:32.868506 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:32.868786 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:32.868789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:32.869161 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:32.869310 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:32.869317 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.870217 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.870220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:32.871151 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:32.871162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.871770 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.871773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:32.871893 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:32.872421 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:32.872440 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:32.875237 (+ 2797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":258,"threads_started":1}
W20260926 03:00:32.875631 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:32.868020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41297 (local address 127.0.97.62:43503)
0926 03:00:32.868235 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:32.868241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:32.868469 (+ 228us) server_negotiation.cc:407] Connection header received
0926 03:00:32.868595 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:32.868598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:32.868673 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:32.868753 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:32.869419 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.870109 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.871259 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:32.871660 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:32.871748 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:32.875411 (+ 3663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41297: BlockingRecv error: recv got EOF from 127.0.97.2:41297 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 03:00:33 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:33 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:33.318233 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:33.310686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:34725)
0926 03:00:33.311205 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:33.311221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:33.311327 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:33.311604 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:33.311607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:33.311935 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:33.312101 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:33.312109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.313132 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.313135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:33.314058 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:33.314068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.314630 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.314634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:33.314771 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:33.315368 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:33.315386 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:33.318020 (+ 2634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":312,"threads_started":1}
W20260926 03:00:33.318411 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:33.310731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34725 (local address 127.0.97.62:43503)
0926 03:00:33.310883 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:33.310888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:33.311288 (+ 400us) server_negotiation.cc:407] Connection header received
0926 03:00:33.311434 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:33.311438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:33.311493 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:33.311568 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:33.312212 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.312945 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.314160 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.314527 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.314609 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:33.318186 (+ 3577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34725: BlockingRecv error: recv got EOF from 127.0.97.3:34725 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:00:33 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:33 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:33.437980 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:33.430436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:34335)
0926 03:00:33.430927 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:33.430943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:33.431045 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:33.431293 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:33.431296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:33.431688 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:33.431836 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:33.431843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.432744 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.432747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:33.433681 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:33.433691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.434271 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.434274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:33.434381 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:33.434965 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:33.434982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:33.437626 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
W20260926 03:00:33.438136 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:33.430528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34335 (local address 127.0.97.62:43503)
0926 03:00:33.430708 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:33.430712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:33.431010 (+ 298us) server_negotiation.cc:407] Connection header received
0926 03:00:33.431122 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:33.431125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:33.431177 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:33.431250 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:33.431952 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.432637 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.433784 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.434164 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.434250 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:33.437804 (+ 3554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34335: BlockingRecv error: recv got EOF from 127.0.97.1:34335 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 03:00:33 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:33 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:33.884431 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:33.876437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:50901)
0926 03:00:33.876890 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:33.876906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:33.877073 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:33.877353 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:33.877356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:33.877699 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:33.877849 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:33.877857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.878999 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.879003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:33.880308 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:33.880320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.880941 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.880945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:33.881113 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:33.881650 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:33.881666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:33.884201 (+ 2535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":217,"threads_started":1}
W20260926 03:00:33.884613 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:33.876621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50901 (local address 127.0.97.62:43503)
0926 03:00:33.876825 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:33.876831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:33.876984 (+ 153us) server_negotiation.cc:407] Connection header received
0926 03:00:33.877172 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:33.877175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:33.877240 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:33.877320 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:33.877960 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.878871 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.880420 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:33.880823 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:33.880914 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:33.884363 (+ 3449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50901: BlockingRecv error: recv got EOF from 127.0.97.2:50901 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:00:34 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:34 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:34.327085 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:34.319214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:43197)
0926 03:00:34.319756 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:34.319772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:34.319878 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:34.320167 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:34.320170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:34.320519 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:34.320673 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:34.320681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.321654 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.321657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:34.322854 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:34.322868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.323703 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.323707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:34.323820 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:34.324351 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:34.324367 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:34.326854 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
W20260926 03:00:34.327215 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:34.319276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43197 (local address 127.0.97.62:43503)
0926 03:00:34.319456 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:34.319462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:34.319851 (+ 389us) server_negotiation.cc:407] Connection header received
0926 03:00:34.319975 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:34.319978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:34.320039 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:34.320123 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:34.320784 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.321548 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.322990 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.323582 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.323675 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:34.327014 (+ 3339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43197: BlockingRecv error: recv got EOF from 127.0.97.3:43197 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 03:00:34 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:34 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:34.446801 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:34.438980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:35177)
0926 03:00:34.439526 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:34.439544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:34.439678 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:34.440121 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:34.440125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:34.440559 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:34.440706 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:34.440715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.441758 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.441761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:34.442655 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:34.442665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.443277 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.443281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:34.443398 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:34.443979 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:34.444002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:34.446601 (+ 2599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":301,"threads_started":1}
W20260926 03:00:34.447036 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:34.439029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35177 (local address 127.0.97.62:43503)
0926 03:00:34.439232 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:34.439238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:34.439794 (+ 556us) server_negotiation.cc:407] Connection header received
0926 03:00:34.439842 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:34.439845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:34.439943 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:34.440043 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:34.440822 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.441632 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.442769 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.443159 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.443245 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:34.446784 (+ 3539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:35177: BlockingRecv error: recv got EOF from 127.0.97.1:35177 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 03:00:34 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:34 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:34.893551 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:34.885414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:44513)
0926 03:00:34.885891 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:34.885906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:34.886068 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:34.886387 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:34.886391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:34.886891 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:34.887085 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:34.887093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.888239 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.888246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:34.889184 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:34.889194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.889880 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.889886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:34.890062 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:34.890596 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:34.890612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:34.893339 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":259,"threads_started":1}
W20260926 03:00:34.893826 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:34.885558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44513 (local address 127.0.97.62:43503)
0926 03:00:34.885759 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:34.885764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:34.885988 (+ 224us) server_negotiation.cc:407] Connection header received
0926 03:00:34.886186 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:34.886191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:34.886260 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:34.886363 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:34.887245 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.888116 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.889325 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:34.889729 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:34.889866 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:34.893551 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44513: BlockingRecv error: recv got EOF from 127.0.97.2:44513 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 03:00:35 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:35 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:35.342334 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:35.328066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:39747)
0926 03:00:35.328513 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:35.328528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:35.328639 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:35.328920 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:35.328923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:35.329332 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:35.329489 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:35.329497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.330465 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.330469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:35.331455 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:35.331466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.332262 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.332268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:35.332467 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:35.339215 (+ 6748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:35.339233 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:35.342012 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":256,"threads_started":1}
W20260926 03:00:35.342475 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:35.328158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39747 (local address 127.0.97.62:43503)
0926 03:00:35.328414 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:35.328420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:35.328605 (+ 185us) server_negotiation.cc:407] Connection header received
0926 03:00:35.328738 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:35.328741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:35.328799 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:35.328877 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:35.329607 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.330345 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.331575 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.332093 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.332217 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:35.342224 (+ 10007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39747: BlockingRecv error: recv got EOF from 127.0.97.3:39747 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 26 03:00:35 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:35 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:35.455953 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:35.447765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:54065)
0926 03:00:35.448327 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:35.448343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:35.448456 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:35.448783 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:35.448786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:35.449268 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:35.449455 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:35.449480 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.450762 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.450766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:35.451652 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:35.451662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.452292 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.452296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:35.452399 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:35.452937 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:35.452953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:35.455713 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":248,"threads_started":1}
W20260926 03:00:35.456085 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:35.447834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54065 (local address 127.0.97.62:43503)
0926 03:00:35.448072 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:35.448081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:35.448428 (+ 347us) server_negotiation.cc:407] Connection header received
0926 03:00:35.448563 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:35.448567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:35.448649 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:35.448754 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:35.449637 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.450626 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.451767 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.452172 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.452264 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:35.455896 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54065: BlockingRecv error: recv got EOF from 127.0.97.1:54065 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 03:00:35 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:35 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:35.903146 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:35.894408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:48665)
0926 03:00:35.895034 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:35.895050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:35.895157 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:35.895440 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:35.895443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:35.895835 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:35.896009 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:35.896017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.897071 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.897076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:35.898103 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:35.898115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.898899 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.898905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:35.899071 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:35.899724 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:35.899752 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:35.902799 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":281,"threads_started":1}
W20260926 03:00:35.903239 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:35.894458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48665 (local address 127.0.97.62:43503)
0926 03:00:35.894612 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:35.894616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:35.895115 (+ 499us) server_negotiation.cc:407] Connection header received
0926 03:00:35.895245 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:35.895249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:35.895313 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:35.895398 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:35.896121 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.896850 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.898229 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:35.898754 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:35.898853 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:35.902986 (+ 4133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48665: BlockingRecv error: recv got EOF from 127.0.97.2:48665 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:00:36 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:36 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:36.350935 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:36.343318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:54649)
0926 03:00:36.343848 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:36.343864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:36.343982 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:36.344257 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:36.344261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:36.344626 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:36.344781 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:36.344789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.345685 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.345689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:36.346659 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:36.346670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.347310 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.347316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:36.347429 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:36.347980 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:36.347997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:36.350688 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":224,"threads_started":1}
W20260926 03:00:36.351090 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:36.343347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54649 (local address 127.0.97.62:43503)
0926 03:00:36.343548 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:36.343555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:36.343945 (+ 390us) server_negotiation.cc:407] Connection header received
0926 03:00:36.344084 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:36.344087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:36.344143 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:36.344217 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:36.344906 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.345587 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.346762 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.347172 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.347262 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:36.350871 (+ 3609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54649: BlockingRecv error: recv got EOF from 127.0.97.3:54649 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 03:00:36 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:36 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:36.465065 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:36.456879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:60335)
0926 03:00:36.457424 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:36.457444 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:36.457569 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:36.457967 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:36.457972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:36.458531 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:36.458755 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:36.458765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.459722 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.459726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:36.460607 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:36.460616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.461259 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.461263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:36.461395 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:36.462019 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:36.462036 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:36.464710 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":301,"threads_started":1}
W20260926 03:00:36.465188 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:36.457121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60335 (local address 127.0.97.62:43503)
0926 03:00:36.457357 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:36.457364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:36.457534 (+ 170us) server_negotiation.cc:407] Connection header received
0926 03:00:36.457735 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:36.457739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:36.457816 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:36.457900 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:36.458881 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.459607 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.460715 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.461118 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.461199 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:36.464885 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60335: BlockingRecv error: recv got EOF from 127.0.97.1:60335 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 03:00:36 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:36 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:36.912807 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:36.904123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:43429)
0926 03:00:36.904609 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:36.904624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:36.904777 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:36.905216 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:36.905221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:36.905602 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:36.905746 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:36.905755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.907549 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.907554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:36.908455 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:36.908466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.909212 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.909217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:36.909333 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:36.909912 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:36.909928 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:36.912599 (+ 2671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
W20260926 03:00:36.913079 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:36.904207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43429 (local address 127.0.97.62:43503)
0926 03:00:36.904389 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:36.904397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:36.904703 (+ 306us) server_negotiation.cc:407] Connection header received
0926 03:00:36.904929 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:36.904934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:36.905036 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:36.905139 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:36.905882 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.907308 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.908590 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:36.909065 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:36.909159 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:36.912787 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43429: BlockingRecv error: recv got EOF from 127.0.97.2:43429 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 03:00:37 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:37 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:37.360458 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:37.351738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:33975)
0926 03:00:37.352197 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:37.352212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:37.352371 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:37.352699 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:37.352704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:37.353136 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:37.353273 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:37.353282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.354294 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.354299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:37.355715 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:37.355730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.356435 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.356440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:37.356635 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:37.357399 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:37.357422 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:37.360174 (+ 2752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":243,"threads_started":1}
W20260926 03:00:37.360526 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:37.351806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33975 (local address 127.0.97.62:43503)
0926 03:00:37.352051 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:37.352057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:37.352295 (+ 238us) server_negotiation.cc:407] Connection header received
0926 03:00:37.352499 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:37.352504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:37.352571 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:37.352672 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:37.353408 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.354150 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.355852 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.356257 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.356352 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:37.360355 (+ 4003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33975: BlockingRecv error: recv got EOF from 127.0.97.3:33975 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 03:00:37 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:37 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:37.473734 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:37.465964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:60003)
0926 03:00:37.466510 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:37.466527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:37.466645 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:37.467036 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:37.467039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:37.467430 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:37.467596 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:37.467604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.468578 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.468581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:37.469520 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:37.469530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.470193 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.470197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:37.470312 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:37.470846 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:37.470862 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:37.473499 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":327,"threads_started":1}
W20260926 03:00:37.473917 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:37.466017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60003 (local address 127.0.97.62:43503)
0926 03:00:37.466172 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:37.466176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:37.466606 (+ 430us) server_negotiation.cc:407] Connection header received
0926 03:00:37.466795 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:37.466802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:37.466891 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:37.466972 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:37.467718 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.468466 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.469635 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.470072 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.470162 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:37.473677 (+ 3515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60003: BlockingRecv error: recv got EOF from 127.0.97.1:60003 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 03:00:37 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:37 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:37.921581 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:37.913800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:42163)
0926 03:00:37.914364 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:37.914385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:37.914541 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:37.914937 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:37.914942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:37.915314 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:37.915477 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:37.915485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.916400 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.916403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:37.917341 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:37.917351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.917992 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.917996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:37.918109 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:37.918698 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:37.918715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:37.921356 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":287,"threads_started":1}
W20260926 03:00:37.921726 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:37.913854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42163 (local address 127.0.97.62:43503)
0926 03:00:37.914032 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:37.914040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:37.914474 (+ 434us) server_negotiation.cc:407] Connection header received
0926 03:00:37.914659 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:37.914664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:37.914762 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:37.914862 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:37.915597 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.916289 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.917459 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:37.917864 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:37.917955 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:37.921525 (+ 3570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42163: BlockingRecv error: recv got EOF from 127.0.97.2:42163 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 03:00:38 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:38 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:38.369302 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:38.361406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:33505)
0926 03:00:38.361915 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:38.361931 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:38.362079 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:38.362414 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:38.362419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:38.362816 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:38.362971 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:38.362980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.363957 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.363964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:38.364885 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:38.364895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.365607 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.365612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:38.365787 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:38.366363 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:38.366381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:38.369066 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":283,"threads_started":1}
W20260926 03:00:38.369460 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:38.361472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33505 (local address 127.0.97.62:43503)
0926 03:00:38.361678 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:38.361686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:38.362016 (+ 330us) server_negotiation.cc:407] Connection header received
0926 03:00:38.362200 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:38.362205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:38.362269 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:38.362391 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:38.363111 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.363811 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.365042 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.365461 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.365595 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:38.369241 (+ 3646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33505: BlockingRecv error: recv got EOF from 127.0.97.3:33505 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 03:00:38 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:38 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:38.483018 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:38.474809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:41893)
0926 03:00:38.475342 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:38.475359 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:38.475507 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:38.475858 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:38.475863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:38.476253 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:38.476394 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:38.476402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.477443 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.477449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:38.478374 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:38.478385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.479109 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.479115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:38.479303 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:38.479905 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:38.479923 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:38.482705 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":297,"threads_started":1}
W20260926 03:00:38.483120 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:38.474966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41893 (local address 127.0.97.62:43503)
0926 03:00:38.475160 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:38.475166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:38.475446 (+ 280us) server_negotiation.cc:407] Connection header received
0926 03:00:38.475630 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:38.475637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:38.475718 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:38.475844 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:38.476539 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.477322 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.478514 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.478942 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.479076 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:38.482890 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41893: BlockingRecv error: recv got EOF from 127.0.97.1:41893 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 03:00:38 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:38 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:38.930339 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:38.922649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:43487)
0926 03:00:38.923154 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:38.923173 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:38.923303 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:38.923645 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:38.923651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:38.924149 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:38.924285 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:38.924293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.925211 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.925214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:38.926099 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:38.926109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.926702 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.926705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:38.926817 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:38.927355 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:38.927371 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:38.930134 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":284,"threads_started":1}
W20260926 03:00:38.930657 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:38.922713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43487 (local address 127.0.97.62:43503)
0926 03:00:38.922923 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:38.922930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:38.923258 (+ 328us) server_negotiation.cc:407] Connection header received
0926 03:00:38.923426 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:38.923431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:38.923512 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:38.923602 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:38.924412 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.925103 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.926214 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:38.926589 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:38.926672 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:38.930299 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43487: BlockingRecv error: recv got EOF from 127.0.97.2:43487 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 03:00:39 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:39 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:39.377588 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:39.370131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:56033)
0926 03:00:39.370595 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:39.370610 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:39.370713 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:39.371017 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:39.371020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:39.371387 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:39.371523 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:39.371531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.372466 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.372470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:39.373447 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:39.373458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.374086 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.374089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:39.374204 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:39.374824 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:39.374846 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:39.377380 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
W20260926 03:00:39.377825 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:39.370336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56033 (local address 127.0.97.62:43503)
0926 03:00:39.370585 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:39.370591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:39.370664 (+ 73us) server_negotiation.cc:407] Connection header received
0926 03:00:39.370835 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:39.370838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:39.370895 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:39.370973 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:39.371643 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.372352 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.373562 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.373965 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.374053 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:39.377566 (+ 3513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56033: BlockingRecv error: recv got EOF from 127.0.97.3:56033 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 26 03:00:39 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:39 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:39.492174 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:39.484075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:47479)
0926 03:00:39.484607 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:39.484623 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:39.484748 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:39.485173 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:39.485178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:39.485579 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:39.485734 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:39.485741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.486742 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.486749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:39.487741 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:39.487752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.488463 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.488470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:39.488636 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:39.489223 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:39.489239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:39.491841 (+ 2602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":273,"threads_started":1}
W20260926 03:00:39.492254 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:39.484161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47479 (local address 127.0.97.62:43503)
0926 03:00:39.484325 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:39.484330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:39.484707 (+ 377us) server_negotiation.cc:407] Connection header received
0926 03:00:39.484909 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:39.484912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:39.484984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:39.485149 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:39.485893 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.486613 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.487873 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.488314 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.488442 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:39.492030 (+ 3588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47479: BlockingRecv error: recv got EOF from 127.0.97.1:47479 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 03:00:39 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:39 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:39.939172 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:39.931323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:53463)
0926 03:00:39.931805 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:39.931821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:39.931927 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:39.932197 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:39.932199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:39.932680 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:39.932902 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:39.932912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.933787 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.933792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:39.934621 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:39.934630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.935258 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.935261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:39.935373 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:39.935970 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:39.935986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:39.938810 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":260,"threads_started":1}
W20260926 03:00:39.939267 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:39.931364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53463 (local address 127.0.97.62:43503)
0926 03:00:39.931557 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:39.931561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:39.931903 (+ 342us) server_negotiation.cc:407] Connection header received
0926 03:00:39.932018 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:39.932021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:39.932078 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:39.932156 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:39.933030 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.933683 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.934728 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:39.935145 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:39.935229 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:39.939009 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53463: BlockingRecv error: recv got EOF from 127.0.97.2:53463 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 03:00:40 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:40 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:40.391537 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:40.378453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:35451)
0926 03:00:40.378941 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:40.378957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:40.379104 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:40.379438 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:40.379443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:40.379854 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:40.380014 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:40.380022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.381088 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.381092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:40.381958 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:40.381968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.382624 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.382632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:40.382794 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:40.388422 (+ 5628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:40.388440 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:40.391197 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":245,"threads_started":1}
W20260926 03:00:40.391594 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:40.378533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35451 (local address 127.0.97.62:43503)
0926 03:00:40.378762 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:40.378767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:40.379040 (+ 273us) server_negotiation.cc:407] Connection header received
0926 03:00:40.379239 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:40.379245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:40.379314 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:40.379413 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:40.380166 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.380909 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.382099 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.382483 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.382613 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:40.391374 (+ 8761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35451: BlockingRecv error: recv got EOF from 127.0.97.3:35451 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 26 03:00:40 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:40 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:40.500845 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:40.493067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:34199)
0926 03:00:40.493587 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:40.493603 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:40.493708 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:40.493986 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:40.493989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:40.494329 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:40.494483 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:40.494491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.495407 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.495410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:40.496331 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:40.496342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.497065 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.497070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:40.497189 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:40.497738 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:40.497755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:40.500630 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
W20260926 03:00:40.501047 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:40.493097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34199 (local address 127.0.97.62:43503)
0926 03:00:40.493266 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:40.493271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:40.493686 (+ 415us) server_negotiation.cc:407] Connection header received
0926 03:00:40.493789 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:40.493792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:40.493863 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:40.493943 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:40.494599 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.495292 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.496457 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.496907 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.497045 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:40.500801 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34199: BlockingRecv error: recv got EOF from 127.0.97.1:34199 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 03:00:40 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:40 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:40.947872 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:40.940062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:53971)
0926 03:00:40.940575 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:40.940590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:40.940692 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:40.940945 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:40.940948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:40.941344 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:40.941501 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:40.941509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.942743 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.942747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:40.943598 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:40.943608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.944211 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.944215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:40.944329 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:40.944841 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:40.944857 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:40.947586 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":277,"threads_started":1}
W20260926 03:00:40.947938 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:40.940149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53971 (local address 127.0.97.62:43503)
0926 03:00:40.940320 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:40.940325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:40.940658 (+ 333us) server_negotiation.cc:407] Connection header received
0926 03:00:40.940769 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:40.940772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:40.940827 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:40.940908 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:40.941615 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.942615 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.943706 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:40.944102 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:40.944189 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:40.947747 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53971: BlockingRecv error: recv got EOF from 127.0.97.2:53971 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 03:00:41 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:41 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:41.400661 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:41.392475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:48349)
0926 03:00:41.392975 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:41.392989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:41.393212 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:41.393557 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:41.393562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:41.393967 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:41.394128 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:41.394137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.395146 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.395152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:41.396178 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:41.396189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.396870 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.396875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:41.397106 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:41.397730 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:41.397747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:41.400367 (+ 2620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":276,"threads_started":1}
W20260926 03:00:41.400786 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:41.392514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48349 (local address 127.0.97.62:43503)
0926 03:00:41.392791 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:41.392798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:41.393140 (+ 342us) server_negotiation.cc:407] Connection header received
0926 03:00:41.393367 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:41.393371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:41.393427 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:41.393527 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:41.394267 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.395007 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.396317 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.396731 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.396860 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:41.400577 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48349: BlockingRecv error: recv got EOF from 127.0.97.3:48349 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 03:00:41 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:41 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:41.509649 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:41.501685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:33567)
0926 03:00:41.502129 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:41.502146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:41.502251 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:41.502537 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:41.502539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:41.502919 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:41.503064 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:41.503071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.504024 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.504028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:41.504967 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:41.504978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.505730 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.505735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:41.505856 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:41.506478 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:41.506500 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:41.509325 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":233,"threads_started":1}
W20260926 03:00:41.509766 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:41.501785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33567 (local address 127.0.97.62:43503)
0926 03:00:41.502009 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:41.502015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:41.502218 (+ 203us) server_negotiation.cc:407] Connection header received
0926 03:00:41.502341 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:41.502345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:41.502421 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:41.502500 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:41.503175 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.503917 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.505142 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.505597 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.505688 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:41.509496 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33567: BlockingRecv error: recv got EOF from 127.0.97.1:33567 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 26 03:00:41 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:41 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:41.956736 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:41.948668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:44233)
0926 03:00:41.949268 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:41.949284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:41.949396 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:41.949664 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:41.949667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:41.950019 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:41.950169 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:41.950176 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.951107 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.951110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:41.952006 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:41.952016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.952670 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.952674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:41.952784 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:41.953443 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:41.953460 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:41.956516 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":321,"threads_started":1}
W20260926 03:00:41.956930 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:41.948825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44233 (local address 127.0.97.62:43503)
0926 03:00:41.948983 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:41.948987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:41.949353 (+ 366us) server_negotiation.cc:407] Connection header received
0926 03:00:41.949488 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:41.949492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:41.949549 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:41.949625 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:41.950285 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.950997 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.952123 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:41.952531 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:41.952616 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:41.956686 (+ 4070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44233: BlockingRecv error: recv got EOF from 127.0.97.2:44233 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 03:00:42 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:42 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:42.409552 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:42.401553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:50859)
0926 03:00:42.402016 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:42.402033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:42.402172 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:42.402522 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:42.402526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:42.402914 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:42.403062 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:42.403090 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.404103 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.404110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:42.405139 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:42.405150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.405859 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.405865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:42.406039 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:42.406651 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:42.406668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:42.409212 (+ 2544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":360,"threads_started":1}
W20260926 03:00:42.409614 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:42.401720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50859 (local address 127.0.97.62:43503)
0926 03:00:42.401904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:42.401911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:42.402103 (+ 192us) server_negotiation.cc:407] Connection header received
0926 03:00:42.402302 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:42.402307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:42.402393 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:42.402492 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:42.403221 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.403972 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.405268 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.405685 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.405825 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:42.409403 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50859: BlockingRecv error: recv got EOF from 127.0.97.3:50859 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 03:00:42 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:42 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:42.518844 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:42.510704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:58427)
0926 03:00:42.511183 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:42.511197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:42.511353 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:42.511687 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:42.511691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:42.512102 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:42.512259 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:42.512267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.513288 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.513294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:42.514179 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:42.514188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.514856 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.514861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:42.515032 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:42.515656 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:42.515671 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:42.518598 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
W20260926 03:00:42.518999 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:42.510770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58427 (local address 127.0.97.62:43503)
0926 03:00:42.511069 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:42.511075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:42.511283 (+ 208us) server_negotiation.cc:407] Connection header received
0926 03:00:42.511472 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:42.511479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:42.511555 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:42.511658 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:42.512394 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.513152 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.514316 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.514722 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.514845 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:42.518775 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58427: BlockingRecv error: recv got EOF from 127.0.97.1:58427 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Sep 26 03:00:42 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:42 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:42.966009 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:42.957658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:36981)
0926 03:00:42.958098 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:42.958113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:42.958270 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:42.958619 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:42.958624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:42.959008 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:42.959186 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:42.959194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.960180 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.960186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:42.961124 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:42.961135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.961840 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.961845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:42.962022 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:42.962766 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:42.962789 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:42.965689 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":231,"threads_started":1}
W20260926 03:00:42.966113 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:42.957745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36981 (local address 127.0.97.62:43503)
0926 03:00:42.957961 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:42.957966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:42.958194 (+ 228us) server_negotiation.cc:407] Connection header received
0926 03:00:42.958399 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:42.958404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:42.958470 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:42.958586 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:42.959345 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.960042 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.961265 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:42.961693 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:42.961831 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:42.965933 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36981: BlockingRecv error: recv got EOF from 127.0.97.2:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 03:00:43 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:43 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:43.418025 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:43.410481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.3:38051)
0926 03:00:43.410961 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:43.410977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:43.411086 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:43.411375 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:43.411379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:43.411723 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:43.411894 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:43.411903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.412834 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.412838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:43.413748 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:43.413758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.414391 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.414395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:43.414504 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:43.415074 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:43.415090 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:43.417774 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":257,"threads_started":1}
W20260926 03:00:43.418164 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:43.410539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38051 (local address 127.0.97.62:43503)
0926 03:00:43.410700 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:43.410704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:43.411045 (+ 341us) server_negotiation.cc:407] Connection header received
0926 03:00:43.411185 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:43.411188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:43.411251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:43.411328 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:43.412012 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.412712 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.413872 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.414276 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.414368 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:43.417974 (+ 3606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38051: BlockingRecv error: recv got EOF from 127.0.97.3:38051 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 03:00:43 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:43 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:43.527868 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:43.519892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.1:40249)
0926 03:00:43.520415 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:43.520431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:43.520535 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:43.520839 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:43.520842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:43.521285 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:43.521471 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:43.521479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.522420 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.522423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:43.523306 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:43.523316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.523977 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.523980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:43.524089 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:43.524720 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:43.524737 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:43.527614 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":300,"threads_started":1}
W20260926 03:00:43.528023 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:43.519953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40249 (local address 127.0.97.62:43503)
0926 03:00:43.520128 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:43.520132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:43.520506 (+ 374us) server_negotiation.cc:407] Connection header received
0926 03:00:43.520630 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:43.520633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:43.520704 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:43.520799 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:43.521603 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.522309 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.523421 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.523864 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.523951 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:43.527797 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40249: BlockingRecv error: recv got EOF from 127.0.97.1:40249 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 03:00:43 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:00:43 dist-test-slave-fccw krb5kdc[8617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:00:43.974859 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:43.967090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43503 (local address 127.0.97.2:38807)
0926 03:00:43.967616 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:43.967632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:43.967732 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:43.967996 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:43.967999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:43.968388 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:43.968567 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:43.968575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.969529 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.969533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:43.970411 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:43.970421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.971067 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.971071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:43.971178 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:43.971727 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:43.971743 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:43.974538 (+ 2795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:43503: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":312,"threads_started":1}
W20260926 03:00:43.974958 9353 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:43.967297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38807 (local address 127.0.97.62:43503)
0926 03:00:43.967448 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:43.967453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:43.967699 (+ 246us) server_negotiation.cc:407] Connection header received
0926 03:00:43.967819 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:43.967823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:43.967877 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:43.967956 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:43.968676 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.969414 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.970518 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:43.970932 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:43.971050 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:43.974765 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38807: BlockingRecv error: recv got EOF from 127.0.97.2:38807 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:00:44.254370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9204
I20260926 03:00:44.266027 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:44.255422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.0.97.1:37431)
0926 03:00:44.255978 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:44.255985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:44.256010 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:00:44.256062 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:44.256068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:44.256290 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:44.256437 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:44.257875 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.259092 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.260237 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.260778 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.260879 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:44.263218 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:44.263226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:44.263238 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:44.263266 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:44.265493 (+ 2227us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:00:44.265732 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60738: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":277,"threads_started":1}
W20260926 03:00:44.266141 9664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60738: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:00:44.266139 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9204: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:37431: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:00:44.266368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 9204
I20260926 03:00:44.282866 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9355
I20260926 03:00:44.294503 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:44.283926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.0.97.2:34187)
0926 03:00:44.284474 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:44.284481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:44.284511 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:00:44.284568 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:44.284574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:44.284786 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:44.284942 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:44.286120 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.287188 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.288536 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.289163 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.289261 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:44.291470 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:44.291478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:44.291491 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:44.291520 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:44.294026 (+ 2506us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:00:44.294282 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46740: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
W20260926 03:00:44.294618 9665 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46740: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:00:44.294631 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9355: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:34187: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:00:44.294750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 9355
I20260926 03:00:44.310667 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9486
I20260926 03:00:44.322603 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:44.311618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.0.97.3:37989)
0926 03:00:44.312098 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:44.312106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:44.312128 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:00:44.312181 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:44.312187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:44.312398 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:44.312598 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:44.313905 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.315489 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.316717 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.317319 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.317447 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:44.319751 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:44.319768 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:44.319781 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:44.319821 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:44.321973 (+ 2152us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 03:00:44.322272 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55056: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
W20260926 03:00:44.322634 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9486: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:37989: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 03:00:44.322722 9666 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55056: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 03:00:44.322809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 9486
I20260926 03:00:44.339028 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9135
I20260926 03:00:44.340217 9197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:44.451632 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 9135
2026-09-26T03:00:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20898 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T03:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:44Z Disabled control of system clock
I20260926 03:00:44.523723 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38089
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:00:44.851440 9671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:44.851703 9671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:44.851779 9671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:44.858825 9671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:44.858919 9671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:44.858955 9671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:44.859030 9671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:44.859091 9671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:44.867602 9671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:44.868739 9671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:44.870450 9671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:44.882225 9679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:44.882302 9677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:44.882328 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:44.882839 9671 server_base.cc:1034] running on GCE node
I20260926 03:00:44.883700 9671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:44.885028 9671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:44.886245 9671 hybrid_clock.cc:648] HybridClock initialized: now 1790391644886190 us; error 46 us; skew 500 ppm
I20260926 03:00:44.886587 9671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:44.888717 9671 webserver.cc:460] Webserver started at http://127.0.97.62:43223/ using document root <none> and password file <none>
I20260926 03:00:44.889460 9671 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:44.889591 9671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:44.889906 9671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:44.892553 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "b34b5ee732b643cc97d1b6a62532e28b"
format_stamp: "Formatted at 2026-09-26 03:00:44 on dist-test-slave-fccw"
server_key: "b6bea0e54f040e73dabd381668e75bf0"
server_key_iv: "34b5f7516cc39d2e85c6342d8793b819"
server_key_version: "encryptionkey@0"
I20260926 03:00:44.893376 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "b34b5ee732b643cc97d1b6a62532e28b"
format_stamp: "Formatted at 2026-09-26 03:00:44 on dist-test-slave-fccw"
server_key: "b6bea0e54f040e73dabd381668e75bf0"
server_key_iv: "34b5f7516cc39d2e85c6342d8793b819"
server_key_version: "encryptionkey@0"
I20260926 03:00:44.898324 9671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:00:44.901556 9685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:44.902791 9671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:00:44.902983 9671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "b34b5ee732b643cc97d1b6a62532e28b"
format_stamp: "Formatted at 2026-09-26 03:00:44 on dist-test-slave-fccw"
server_key: "b6bea0e54f040e73dabd381668e75bf0"
server_key_iv: "34b5f7516cc39d2e85c6342d8793b819"
server_key_version: "encryptionkey@0"
I20260926 03:00:44.903188 9671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:44.921607 9671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:44.922317 9671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:44.922544 9671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:44.939868 9671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38089
I20260926 03:00:44.939899 9736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38089 every 8 connection(s)
I20260926 03:00:44.941253 9671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:44.944630 9737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:44.951578 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 9671
I20260926 03:00:44.951768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:44.952389 388 external_mini_cluster.cc:1428] Setting key 9c948acf652e2459f097123c42cd71da
I20260926 03:00:44.954246 9737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b: Bootstrap starting.
I20260926 03:00:44.957227 9737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:44.958457 9737 log.cc:824] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:44.961578 9737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b: No bootstrap required, opened a new log
I20260926 03:00:44.965533 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:44.954302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.0.97.62:38089)
0926 03:00:44.955279 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:44.955289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:44.955327 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:00:44.956883 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:44.956904 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:44.957460 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:44.957781 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:44.958393 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.959752 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.961455 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:44.962129 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:44.962246 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:44.963402 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:44.963411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:44.963424 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:44.963942 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:44.964012 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:44.964033 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:44.964127 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:44.964350 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 03:00:44.964649 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":367,"threads_started":1}
I20260926 03:00:44.969761 9737 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34b5ee732b643cc97d1b6a62532e28b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38089 } }
I20260926 03:00:44.970099 9737 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:44.970216 9737 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34b5ee732b643cc97d1b6a62532e28b, State: Initialized, Role: FOLLOWER
I20260926 03:00:44.970800 9737 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34b5ee732b643cc97d1b6a62532e28b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38089 } }
I20260926 03:00:44.971127 9737 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:44.971290 9737 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:44.971463 9737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:44.973842 9737 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34b5ee732b643cc97d1b6a62532e28b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38089 } }
I20260926 03:00:44.974324 9737 leader_election.cc:302] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34b5ee732b643cc97d1b6a62532e28b; no voters:
I20260926 03:00:44.974853 9737 leader_election.cc:288] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:44.975004 9742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:44.975798 9742 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [term 1 LEADER]: Becoming Leader. State: Replica: b34b5ee732b643cc97d1b6a62532e28b, State: Running, Role: LEADER
I20260926 03:00:44.976346 9742 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34b5ee732b643cc97d1b6a62532e28b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38089 } }
I20260926 03:00:44.976883 9737 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:44.979383 9744 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34b5ee732b643cc97d1b6a62532e28b. Latest consensus state: current_term: 1 leader_uuid: "b34b5ee732b643cc97d1b6a62532e28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34b5ee732b643cc97d1b6a62532e28b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38089 } } }
I20260926 03:00:44.979722 9744 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:44.979480 9743 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34b5ee732b643cc97d1b6a62532e28b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34b5ee732b643cc97d1b6a62532e28b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38089 } } }
I20260926 03:00:44.979873 9743 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:44.981459 9748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:44.986250 9748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:44.997512 9748 catalog_manager.cc:1347] Generated new cluster ID: d91f160680f8487a8ae2ecffde0f5db2
I20260926 03:00:44.997627 9748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:45.009552 9748 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:45.010764 9748 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:45.016834 9748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34b5ee732b643cc97d1b6a62532e28b: Generated new TSK 0
I20260926 03:00:45.017599 9748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:00:45.027426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38089
--builtin_ntp_servers=127.0.97.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 03:00:45.374444 9761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:45.374706 9761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:45.374786 9761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:45.381989 9761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:45.382094 9761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:45.382206 9761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:45.390939 9761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:45.392062 9761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:45.393841 9761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:45.406653 9766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:45.406713 9769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:45.406734 9767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:45.407496 9761 server_base.cc:1034] running on GCE node
I20260926 03:00:45.408113 9761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:45.408916 9761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:45.410104 9761 hybrid_clock.cc:648] HybridClock initialized: now 1790391645410040 us; error 59 us; skew 500 ppm
I20260926 03:00:45.410406 9761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:45.425266 9761 webserver.cc:460] Webserver started at http://127.0.97.1:45385/ using document root <none> and password file <none>
I20260926 03:00:45.425966 9761 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:45.426096 9761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:45.426429 9761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:45.429211 9761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "d44585f9aa364874b6e1aa8b534cb7ba"
format_stamp: "Formatted at 2026-09-26 03:00:45 on dist-test-slave-fccw"
server_key: "102daa95b7ebf5e1ee5b655a8591585b"
server_key_iv: "6a827a51e97e0c9cac3986706818b47d"
server_key_version: "encryptionkey@0"
I20260926 03:00:45.429978 9761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d44585f9aa364874b6e1aa8b534cb7ba"
format_stamp: "Formatted at 2026-09-26 03:00:45 on dist-test-slave-fccw"
server_key: "102daa95b7ebf5e1ee5b655a8591585b"
server_key_iv: "6a827a51e97e0c9cac3986706818b47d"
server_key_version: "encryptionkey@0"
I20260926 03:00:45.434909 9761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:00:45.438071 9775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:45.439308 9761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:00:45.439473 9761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "d44585f9aa364874b6e1aa8b534cb7ba"
format_stamp: "Formatted at 2026-09-26 03:00:45 on dist-test-slave-fccw"
server_key: "102daa95b7ebf5e1ee5b655a8591585b"
server_key_iv: "6a827a51e97e0c9cac3986706818b47d"
server_key_version: "encryptionkey@0"
I20260926 03:00:45.439674 9761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:45.458580 9761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:45.459348 9761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:45.459615 9761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:45.460477 9761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:45.461912 9761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:45.462039 9761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:45.462157 9761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:45.462251 9761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:45.494457 9761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44211
I20260926 03:00:45.494508 9887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44211 every 8 connection(s)
I20260926 03:00:45.495747 9761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:45.504240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 9761
I20260926 03:00:45.504431 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:00:45.505095 388 external_mini_cluster.cc:1428] Setting key 3a0780bf9dc1dfcbc4714f70afbb7271
I20260926 03:00:45.508021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38089
--builtin_ntp_servers=127.0.97.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:00:45.508126 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:45.498229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36587 (local address 127.0.97.62:38089)
0926 03:00:45.498663 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:45.498668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:45.499719 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 03:00:45.501461 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:45.501466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:45.501556 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:45.501673 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:45.502690 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.503412 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.505170 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.505559 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.505717 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:45.507278 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:45.507283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:45.507287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:45.507306 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:45.507352 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:45.507366 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:45.507448 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:45.507658 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 03:00:45.507748 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":234,"threads_started":1}
I20260926 03:00:45.509145 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:45.498680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38089 (local address 127.0.97.1:36587)
0926 03:00:45.499558 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:45.499599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:45.501210 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:45.501895 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:45.501903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:45.501983 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:45.502534 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:45.502548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.503566 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.503570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:45.504986 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:45.505040 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.505821 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.505829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:45.506145 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:45.506908 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:00:45.506911 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:45.506961 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:00:45.507064 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:45.507465 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:45.507472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:45.507478 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:00:45.507610 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 03:00:45.507989 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260926 03:00:45.510684 9888 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38089
I20260926 03:00:45.510957 9888 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:45.511688 9888 heartbeater.cc:511] Master 127.0.97.62:38089 requested a full tablet report, sending...
I20260926 03:00:45.513831 9702 ts_manager.cc:194] Registered new tserver with Master: d44585f9aa364874b6e1aa8b534cb7ba (127.0.97.1:44211)
I20260926 03:00:45.515380 9702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36587
W20260926 03:00:45.818805 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:45.819124 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:45.819267 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:45.826202 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:45.826283 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:45.826453 9892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:45.835000 9892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:45.836248 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:45.837992 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:45.850504 9897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:45.850553 9898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:45.850730 9900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:45.851323 9892 server_base.cc:1034] running on GCE node
I20260926 03:00:45.851939 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:45.852787 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:45.853973 9892 hybrid_clock.cc:648] HybridClock initialized: now 1790391645853927 us; error 40 us; skew 500 ppm
I20260926 03:00:45.854274 9892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:45.856336 9892 webserver.cc:460] Webserver started at http://127.0.97.2:41181/ using document root <none> and password file <none>
I20260926 03:00:45.856980 9892 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:45.857151 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:45.857474 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:45.860152 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "1d4447c6f51f4a33a16fa3eaaead1c5a"
format_stamp: "Formatted at 2026-09-26 03:00:45 on dist-test-slave-fccw"
server_key: "08d9f78901c13043a1c77db2e02f708f"
server_key_iv: "e5ef9b860d67d3116e0d853626b77854"
server_key_version: "encryptionkey@0"
I20260926 03:00:45.860903 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1d4447c6f51f4a33a16fa3eaaead1c5a"
format_stamp: "Formatted at 2026-09-26 03:00:45 on dist-test-slave-fccw"
server_key: "08d9f78901c13043a1c77db2e02f708f"
server_key_iv: "e5ef9b860d67d3116e0d853626b77854"
server_key_version: "encryptionkey@0"
I20260926 03:00:45.865640 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:00:45.868763 9906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:45.870060 9892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:00:45.870235 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "1d4447c6f51f4a33a16fa3eaaead1c5a"
format_stamp: "Formatted at 2026-09-26 03:00:45 on dist-test-slave-fccw"
server_key: "08d9f78901c13043a1c77db2e02f708f"
server_key_iv: "e5ef9b860d67d3116e0d853626b77854"
server_key_version: "encryptionkey@0"
I20260926 03:00:45.870450 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:45.892573 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:45.893463 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:45.893743 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:45.894658 9892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:45.896054 9892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:45.896170 9892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:45.896327 9892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:45.896417 9892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:45.937438 9892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41553
I20260926 03:00:45.937757 10018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41553 every 8 connection(s)
I20260926 03:00:45.938786 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:45.940958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 9892
I20260926 03:00:45.941186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:00:45.941908 388 external_mini_cluster.cc:1428] Setting key 22f3dda32beb1a698bed5798ca055aa5
I20260926 03:00:45.946216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38089
--builtin_ntp_servers=127.0.97.20:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:00:45.975379 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:45.960084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36095 (local address 127.0.97.62:38089)
0926 03:00:45.960327 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:45.960336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:45.966018 (+ 5682us) server_negotiation.cc:407] Connection header received
0926 03:00:45.967728 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:45.967733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:45.967819 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:45.967920 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:45.969420 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.970162 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.971707 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.972409 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.972547 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:45.974350 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:45.974358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:45.974366 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:45.974401 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:45.974443 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:45.974462 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:45.974556 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:45.974919 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 03:00:45.975020 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:00:45.976883 10022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:45.959403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38089 (local address 127.0.97.2:36095)
0926 03:00:45.965860 (+ 6457us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:45.965908 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:45.967461 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:45.968336 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:45.968348 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:45.968474 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:45.969252 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:45.969271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.970324 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.970329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:45.971549 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:45.971560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:45.972679 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:45.972689 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:45.973069 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:45.973954 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:00:45.973960 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:45.974017 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:00:45.974154 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:45.974581 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:45.974595 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:45.974628 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:00:45.974891 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 03:00:45.975342 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5903,"thread_start_us":5653,"threads_started":1}
I20260926 03:00:45.978677 10019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38089
I20260926 03:00:45.979061 10019 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:45.980086 10019 heartbeater.cc:511] Master 127.0.97.62:38089 requested a full tablet report, sending...
I20260926 03:00:45.981786 9702 ts_manager.cc:194] Registered new tserver with Master: 1d4447c6f51f4a33a16fa3eaaead1c5a (127.0.97.2:41553)
I20260926 03:00:45.983136 9702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36095
W20260926 03:00:46.286482 10021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:46.286773 10021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:46.286931 10021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:46.293817 10021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:46.293934 10021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:46.294099 10021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:46.302729 10021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:46.303938 10021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:46.305698 10021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:46.318662 10027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:46.318714 10028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:46.318684 10030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:46.319602 10021 server_base.cc:1034] running on GCE node
I20260926 03:00:46.320137 10021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:46.321069 10021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:46.322294 10021 hybrid_clock.cc:648] HybridClock initialized: now 1790391646322246 us; error 39 us; skew 500 ppm
I20260926 03:00:46.322609 10021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:46.324651 10021 webserver.cc:460] Webserver started at http://127.0.97.3:37003/ using document root <none> and password file <none>
I20260926 03:00:46.325357 10021 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:46.325484 10021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:46.325798 10021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:46.328274 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "6c4a1e9bae544e56a8bb391de55e6b83"
format_stamp: "Formatted at 2026-09-26 03:00:46 on dist-test-slave-fccw"
server_key: "2717c6c135fff436a9c86f7fbd0563df"
server_key_iv: "0d331e542192c8429be1a3e01e59f8f7"
server_key_version: "encryptionkey@0"
I20260926 03:00:46.328990 10021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6c4a1e9bae544e56a8bb391de55e6b83"
format_stamp: "Formatted at 2026-09-26 03:00:46 on dist-test-slave-fccw"
server_key: "2717c6c135fff436a9c86f7fbd0563df"
server_key_iv: "0d331e542192c8429be1a3e01e59f8f7"
server_key_version: "encryptionkey@0"
I20260926 03:00:46.333865 10021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:00:46.337086 10036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:46.338265 10021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:00:46.338438 10021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "6c4a1e9bae544e56a8bb391de55e6b83"
format_stamp: "Formatted at 2026-09-26 03:00:46 on dist-test-slave-fccw"
server_key: "2717c6c135fff436a9c86f7fbd0563df"
server_key_iv: "0d331e542192c8429be1a3e01e59f8f7"
server_key_version: "encryptionkey@0"
I20260926 03:00:46.338627 10021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:46.359648 10021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:46.360459 10021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:46.360743 10021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:46.361685 10021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:46.363075 10021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:46.363190 10021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:46.363322 10021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:46.363409 10021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:46.398407 10021 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39387
I20260926 03:00:46.398512 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39387 every 8 connection(s)
I20260926 03:00:46.399719 10021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:46.408412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10021
I20260926 03:00:46.408605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:00:46.409174 388 external_mini_cluster.cc:1428] Setting key 0d3deceb1fd5de1c83e24555972f49f5
I20260926 03:00:46.412985 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:46.402289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59911 (local address 127.0.97.62:38089)
0926 03:00:46.402474 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:46.402478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:46.404147 (+ 1669us) server_negotiation.cc:407] Connection header received
0926 03:00:46.405905 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:46.405909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:46.405974 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:46.406193 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:46.407164 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.407955 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.410092 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.410656 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.410785 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:46.412218 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:46.412223 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:46.412226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:46.412242 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:46.412277 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:46.412289 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:46.412382 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:46.412594 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 03:00:46.412678 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:00:46.414011 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:46.402846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38089 (local address 127.0.97.3:59911)
0926 03:00:46.403944 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:46.404007 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:46.405663 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:46.406340 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:46.406348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:46.406469 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:46.407017 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:46.407033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.408137 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.408144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:46.409967 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:46.409979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.410890 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.410897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:46.411214 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:46.411907 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:00:46.411910 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:46.411947 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:00:46.412024 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:46.412381 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:46.412393 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:46.412403 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:00:46.412585 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 03:00:46.412969 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":381,"threads_started":1}
I20260926 03:00:46.415274 10149 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38089
I20260926 03:00:46.415534 10149 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:46.416236 10149 heartbeater.cc:511] Master 127.0.97.62:38089 requested a full tablet report, sending...
I20260926 03:00:46.417454 9702 ts_manager.cc:194] Registered new tserver with Master: 6c4a1e9bae544e56a8bb391de55e6b83 (127.0.97.3:39387)
I20260926 03:00:46.418155 9702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59911
I20260926 03:00:46.424122 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:00:46.429876 9890 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:46.428351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.0.97.62:38089)
0926 03:00:46.428523 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:46.428527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:46.429041 (+ 514us) server_negotiation.cc:407] Connection header received
0926 03:00:46.429192 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:46.429214 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:46.429291 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:46.429377 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:46.429721 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32942: BlockingRecv error: recv got EOF from 127.0.0.1:32942 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:00:46.430452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9761
I20260926 03:00:46.437726 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:46.431301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.0.97.1:44211)
0926 03:00:46.431821 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:46.431828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:46.431850 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:00:46.431923 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:46.431929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:46.432108 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:46.432220 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:46.432575 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.433564 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.434721 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.435219 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.435327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:46.436257 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:46.436265 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:46.436277 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:46.436798 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:46.436839 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:46.436857 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:46.436939 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:46.437235 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:00:46.437366 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
I20260926 03:00:46.438879 9883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:46.518885 9888 heartbeater.cc:503] Master 127.0.97.62:38089 was elected leader, sending a full tablet report...
I20260926 03:00:46.519764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 9761
W20260926 03:00:46.535938 9689 connection.cc:535] server connection from 127.0.97.1:36587 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:00:46.536182 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9892
I20260926 03:00:46.544206 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:46.537116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.0.97.2:41553)
0926 03:00:46.537584 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:46.537592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:46.537616 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:00:46.537673 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:46.537679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:46.537905 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:46.538081 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:46.538545 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.539546 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.540830 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.541490 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.541650 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:46.542718 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:46.542735 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:46.542749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:46.543361 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:46.543419 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:46.543445 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:46.543563 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:46.543743 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 03:00:46.543870 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260926 03:00:46.545449 10014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:46.624787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 9892
I20260926 03:00:46.641340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10021
I20260926 03:00:46.648903 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:46.642290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.0.97.3:39387)
0926 03:00:46.642909 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:46.642916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:46.642945 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:00:46.643001 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:46.643007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:46.643208 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:46.643346 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:46.643701 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.644781 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.646020 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:46.646560 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:46.646659 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:46.647659 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:46.647668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:46.647681 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:46.648216 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:46.648275 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:46.648293 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:46.648374 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:46.648509 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 03:00:46.648629 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":294,"threads_started":1}
I20260926 03:00:46.650192 10144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:46.730268 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10021
I20260926 03:00:46.745653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 9671
I20260926 03:00:46.746832 9732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:46.845258 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 9671
2026-09-26T03:00:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2389 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T03:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:46Z Disabled control of system clock
I20260926 03:00:46.913398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35549
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:00:47.228273 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:47.228534 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:47.228613 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:47.235402 10168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:47.235494 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:47.235530 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:47.235584 10168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:47.235620 10168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:47.244195 10168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:47.245340 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:47.246930 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:47.258245 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:47.258284 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:47.258309 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:47.259308 10168 server_base.cc:1034] running on GCE node
I20260926 03:00:47.260040 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:47.261320 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:47.262518 10168 hybrid_clock.cc:648] HybridClock initialized: now 1790391647262484 us; error 25 us; skew 500 ppm
I20260926 03:00:47.262801 10168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:47.264827 10168 webserver.cc:460] Webserver started at http://127.0.97.62:44301/ using document root <none> and password file <none>
I20260926 03:00:47.265554 10168 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:47.265640 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:47.265877 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:47.268473 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "bfdbb8831c4a448c9e5ea39aad372256"
format_stamp: "Formatted at 2026-09-26 03:00:47 on dist-test-slave-fccw"
server_key: "c4efc20dae10ff470249e41ae24042f4"
server_key_iv: "de1766a375a2affe1a06f645fa3386a1"
server_key_version: "encryptionkey@0"
I20260926 03:00:47.269280 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "bfdbb8831c4a448c9e5ea39aad372256"
format_stamp: "Formatted at 2026-09-26 03:00:47 on dist-test-slave-fccw"
server_key: "c4efc20dae10ff470249e41ae24042f4"
server_key_iv: "de1766a375a2affe1a06f645fa3386a1"
server_key_version: "encryptionkey@0"
I20260926 03:00:47.274380 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 03:00:47.277531 10182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:47.278805 10168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:00:47.278977 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "bfdbb8831c4a448c9e5ea39aad372256"
format_stamp: "Formatted at 2026-09-26 03:00:47 on dist-test-slave-fccw"
server_key: "c4efc20dae10ff470249e41ae24042f4"
server_key_iv: "de1766a375a2affe1a06f645fa3386a1"
server_key_version: "encryptionkey@0"
I20260926 03:00:47.279187 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:47.284670 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:47.285444 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:47.285723 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:47.303050 10168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35549
I20260926 03:00:47.303082 10233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35549 every 8 connection(s)
I20260926 03:00:47.304323 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:47.307850 10234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:47.310787 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10168
I20260926 03:00:47.311097 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:47.311717 388 external_mini_cluster.cc:1428] Setting key eec5e827843ad56d2863ce30c86a68de
I20260926 03:00:47.319710 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:47.313644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.97.62:35549)
0926 03:00:47.314595 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:47.314605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:47.314645 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:00:47.316222 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:47.316252 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:47.316690 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:47.317107 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:47.318146 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:47.318152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:47.318164 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:47.318176 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:47.318238 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:47.318263 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:47.318475 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:47.318603 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 03:00:47.318827 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":349,"threads_started":1}
I20260926 03:00:47.320082 10234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256: Bootstrap starting.
I20260926 03:00:47.323364 10234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:47.324256 10234 log.cc:824] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:47.326786 10234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256: No bootstrap required, opened a new log
I20260926 03:00:47.333966 10234 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfdbb8831c4a448c9e5ea39aad372256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35549 } }
I20260926 03:00:47.334335 10234 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:47.334446 10234 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfdbb8831c4a448c9e5ea39aad372256, State: Initialized, Role: FOLLOWER
I20260926 03:00:47.334966 10234 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfdbb8831c4a448c9e5ea39aad372256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35549 } }
I20260926 03:00:47.335208 10234 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:47.335346 10234 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:47.335510 10234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:47.337643 10234 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfdbb8831c4a448c9e5ea39aad372256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35549 } }
I20260926 03:00:47.338104 10234 leader_election.cc:302] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfdbb8831c4a448c9e5ea39aad372256; no voters:
I20260926 03:00:47.338663 10234 leader_election.cc:288] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:47.338800 10239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:47.339607 10239 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [term 1 LEADER]: Becoming Leader. State: Replica: bfdbb8831c4a448c9e5ea39aad372256, State: Running, Role: LEADER
I20260926 03:00:47.340173 10239 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfdbb8831c4a448c9e5ea39aad372256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35549 } }
I20260926 03:00:47.341090 10234 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:47.342967 10241 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfdbb8831c4a448c9e5ea39aad372256. Latest consensus state: current_term: 1 leader_uuid: "bfdbb8831c4a448c9e5ea39aad372256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfdbb8831c4a448c9e5ea39aad372256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35549 } } }
I20260926 03:00:47.342991 10240 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfdbb8831c4a448c9e5ea39aad372256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfdbb8831c4a448c9e5ea39aad372256" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35549 } } }
I20260926 03:00:47.343391 10241 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:47.343466 10240 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:47.345247 10245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:47.349951 10245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:47.359378 10245 catalog_manager.cc:1347] Generated new cluster ID: baf63e27f97643a7a902ace3c030454c
I20260926 03:00:47.359512 10245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:47.388358 10245 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:47.388818 10245 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:47.397491 10245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfdbb8831c4a448c9e5ea39aad372256: Generated new TSK 0
I20260926 03:00:47.398264 10245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:00:47.405444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35549
--builtin_ntp_servers=127.0.97.20:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:00:47.724082 10258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:47.724339 10258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:47.724529 10258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:47.731586 10258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:47.731675 10258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:47.731840 10258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:47.740468 10258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:47.741838 10258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:47.743630 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:47.756336 10263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:47.756306 10266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:47.756371 10264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:47.757275 10258 server_base.cc:1034] running on GCE node
I20260926 03:00:47.757762 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:47.758559 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:47.759790 10258 hybrid_clock.cc:648] HybridClock initialized: now 1790391647759743 us; error 43 us; skew 500 ppm
I20260926 03:00:47.760090 10258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:47.762151 10258 webserver.cc:460] Webserver started at http://127.0.97.1:35741/ using document root <none> and password file <none>
I20260926 03:00:47.762840 10258 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:47.762964 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:47.763268 10258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:47.766072 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "c70879511659465e8af10b08c5022d16"
format_stamp: "Formatted at 2026-09-26 03:00:47 on dist-test-slave-fccw"
server_key: "997aad3897c4d0d9fe4f4a825d9e43d8"
server_key_iv: "f084c5c0f12fec23296c620c5baf1543"
server_key_version: "encryptionkey@0"
I20260926 03:00:47.766876 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c70879511659465e8af10b08c5022d16"
format_stamp: "Formatted at 2026-09-26 03:00:47 on dist-test-slave-fccw"
server_key: "997aad3897c4d0d9fe4f4a825d9e43d8"
server_key_iv: "f084c5c0f12fec23296c620c5baf1543"
server_key_version: "encryptionkey@0"
I20260926 03:00:47.771752 10258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 03:00:47.774936 10272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:47.776140 10258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:00:47.776298 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "c70879511659465e8af10b08c5022d16"
format_stamp: "Formatted at 2026-09-26 03:00:47 on dist-test-slave-fccw"
server_key: "997aad3897c4d0d9fe4f4a825d9e43d8"
server_key_iv: "f084c5c0f12fec23296c620c5baf1543"
server_key_version: "encryptionkey@0"
I20260926 03:00:47.776515 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:47.781958 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:47.782707 10258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:47.782919 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:47.783771 10258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:47.785058 10258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:47.785147 10258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:47.785223 10258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:47.785275 10258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:47.816833 10258 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34629
I20260926 03:00:47.816880 10384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34629 every 8 connection(s)
I20260926 03:00:47.818272 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:47.825860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10258
I20260926 03:00:47.826084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:00:47.826309 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:47.820793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41345 (local address 127.0.97.62:35549)
0926 03:00:47.821250 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:47.821255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:47.822307 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 03:00:47.823939 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:47.823945 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:47.824017 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:47.824142 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:47.825567 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:47.825571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:47.825574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:47.825588 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:47.825625 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:47.825635 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:47.825735 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:47.825983 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 03:00:47.826050 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":276,"threads_started":1}
I20260926 03:00:47.826844 388 external_mini_cluster.cc:1428] Setting key b3508712bdeefaf3d46560a877b469f2
I20260926 03:00:47.827427 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:47.821300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35549 (local address 127.0.97.1:41345)
0926 03:00:47.822160 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:47.822195 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:47.823653 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:47.824366 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:47.824374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:47.824459 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:47.825242 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:00:47.825244 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:47.825278 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:00:47.825355 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:47.825738 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:47.825749 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:47.825758 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:00:47.826035 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 03:00:47.826277 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":267,"threads_started":1}
I20260926 03:00:47.829095 10385 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35549
I20260926 03:00:47.829409 10385 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:47.829659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35549
--builtin_ntp_servers=127.0.97.20:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:00:47.830051 10385 heartbeater.cc:511] Master 127.0.97.62:35549 requested a full tablet report, sending...
I20260926 03:00:47.832255 10198 ts_manager.cc:194] Registered new tserver with Master: c70879511659465e8af10b08c5022d16 (127.0.97.1:34629)
W20260926 03:00:48.142931 10389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:48.143231 10389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:48.143383 10389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:48.150542 10389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:48.150667 10389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:48.150838 10389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:48.159628 10389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:48.160915 10389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:48.162652 10389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:48.175565 10394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:48.175611 10397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:48.175637 10395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:48.176447 10389 server_base.cc:1034] running on GCE node
I20260926 03:00:48.177085 10389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:48.177968 10389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:48.179194 10389 hybrid_clock.cc:648] HybridClock initialized: now 1790391648179147 us; error 59 us; skew 500 ppm
I20260926 03:00:48.179495 10389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:48.181744 10389 webserver.cc:460] Webserver started at http://127.0.97.2:38467/ using document root <none> and password file <none>
I20260926 03:00:48.182431 10389 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:48.182557 10389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:48.182855 10389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:48.185634 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "56a1fa9ae9184526b9ad22602c64cdf1"
format_stamp: "Formatted at 2026-09-26 03:00:48 on dist-test-slave-fccw"
server_key: "67e00a77b08eae5af97a4f689adecba3"
server_key_iv: "43d656c074cd26936aa02e0715249c63"
server_key_version: "encryptionkey@0"
I20260926 03:00:48.186398 10389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "56a1fa9ae9184526b9ad22602c64cdf1"
format_stamp: "Formatted at 2026-09-26 03:00:48 on dist-test-slave-fccw"
server_key: "67e00a77b08eae5af97a4f689adecba3"
server_key_iv: "43d656c074cd26936aa02e0715249c63"
server_key_version: "encryptionkey@0"
I20260926 03:00:48.191164 10389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:00:48.194332 10403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:48.195477 10389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:00:48.195647 10389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "56a1fa9ae9184526b9ad22602c64cdf1"
format_stamp: "Formatted at 2026-09-26 03:00:48 on dist-test-slave-fccw"
server_key: "67e00a77b08eae5af97a4f689adecba3"
server_key_iv: "43d656c074cd26936aa02e0715249c63"
server_key_version: "encryptionkey@0"
I20260926 03:00:48.195869 10389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:48.201058 10389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:48.201790 10389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:48.202050 10389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:48.202877 10389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:48.204286 10389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:48.204401 10389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:48.204527 10389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:48.204619 10389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:48.235919 10389 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43827
I20260926 03:00:48.235966 10515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43827 every 8 connection(s)
I20260926 03:00:48.237188 10389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:48.241286 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10389
I20260926 03:00:48.241495 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:00:48.241907 388 external_mini_cluster.cc:1428] Setting key 4dca205d9aa48470d3506542b0f4e189
I20260926 03:00:48.244671 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35549
--builtin_ntp_servers=127.0.97.20:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:00:48.245623 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:48.239661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42271 (local address 127.0.97.62:35549)
0926 03:00:48.239868 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:48.239874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:48.241338 (+ 1464us) server_negotiation.cc:407] Connection header received
0926 03:00:48.243325 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:48.243330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:48.243405 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:48.243521 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:48.244769 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:48.244775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:48.244781 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:48.244800 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:48.244837 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:48.244853 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:48.244958 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:48.245271 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:00:48.245336 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:00:48.246532 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:48.240188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35549 (local address 127.0.97.2:42271)
0926 03:00:48.241184 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:48.241226 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:48.243071 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:48.243746 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:48.243754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:48.243844 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:48.244488 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:00:48.244490 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:48.244525 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:00:48.244613 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:48.245046 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:48.245056 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:48.245064 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:00:48.245352 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 03:00:48.245602 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":272,"threads_started":1}
I20260926 03:00:48.247874 10516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35549
I20260926 03:00:48.248147 10516 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:48.248668 10516 heartbeater.cc:511] Master 127.0.97.62:35549 requested a full tablet report, sending...
I20260926 03:00:48.250072 10198 ts_manager.cc:194] Registered new tserver with Master: 56a1fa9ae9184526b9ad22602c64cdf1 (127.0.97.2:43827)
W20260926 03:00:48.565572 10519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:48.565836 10519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:48.565924 10519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:48.572922 10519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:48.573062 10519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:48.573189 10519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:48.582280 10519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:48.583650 10519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:48.585510 10519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:48.598243 10524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:48.598281 10525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:48.598243 10527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:48.602351 10519 server_base.cc:1034] running on GCE node
I20260926 03:00:48.602887 10519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:48.603749 10519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:48.604962 10519 hybrid_clock.cc:648] HybridClock initialized: now 1790391648604915 us; error 26 us; skew 500 ppm
I20260926 03:00:48.605419 10519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:00:48.607554 10519 webserver.cc:460] Webserver started at http://127.0.97.3:43491/ using document root <none> and password file <none>
I20260926 03:00:48.608240 10519 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:48.608371 10519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:48.608742 10519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:48.611713 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "bb4f143bec704e5e9691116292426168"
format_stamp: "Formatted at 2026-09-26 03:00:48 on dist-test-slave-fccw"
server_key: "293f190e7e8e10c17eff9c9438e907fd"
server_key_iv: "daa9318a4d4353742d9b2474efe84b6b"
server_key_version: "encryptionkey@0"
I20260926 03:00:48.612411 10519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bb4f143bec704e5e9691116292426168"
format_stamp: "Formatted at 2026-09-26 03:00:48 on dist-test-slave-fccw"
server_key: "293f190e7e8e10c17eff9c9438e907fd"
server_key_iv: "daa9318a4d4353742d9b2474efe84b6b"
server_key_version: "encryptionkey@0"
I20260926 03:00:48.617470 10519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:00:48.620637 10533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:48.621989 10519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:00:48.622198 10519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "bb4f143bec704e5e9691116292426168"
format_stamp: "Formatted at 2026-09-26 03:00:48 on dist-test-slave-fccw"
server_key: "293f190e7e8e10c17eff9c9438e907fd"
server_key_iv: "daa9318a4d4353742d9b2474efe84b6b"
server_key_version: "encryptionkey@0"
I20260926 03:00:48.622414 10519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:48.627911 10519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:48.628670 10519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:48.628926 10519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:48.629829 10519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:48.631150 10519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:48.631273 10519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:48.631403 10519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:48.631490 10519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:48.662882 10519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36875
I20260926 03:00:48.662930 10645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36875 every 8 connection(s)
I20260926 03:00:48.664222 10519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:48.666105 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10519
I20260926 03:00:48.666304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:00:48.666723 388 external_mini_cluster.cc:1428] Setting key 0315332454a43aeb54d5b6be12c32dd7
I20260926 03:00:48.673094 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:48.666982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35197 (local address 127.0.97.62:35549)
0926 03:00:48.667214 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:48.667222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:48.668964 (+ 1742us) server_negotiation.cc:407] Connection header received
0926 03:00:48.670857 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:48.670862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:48.670926 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:48.671039 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:48.672352 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:48.672357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:48.672362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:48.672379 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:48.672416 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:48.672433 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:48.672556 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:48.672745 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 03:00:48.672808 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:00:48.673969 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:48.667659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35549 (local address 127.0.97.3:35197)
0926 03:00:48.668812 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:48.668857 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:48.670594 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:48.671281 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:48.671289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:48.671376 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:48.672072 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:00:48.672074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:48.672111 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:00:48.672185 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:48.672525 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:48.672537 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:48.672545 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:00:48.672823 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 03:00:48.673087 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":363,"threads_started":1}
I20260926 03:00:48.675191 10646 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35549
I20260926 03:00:48.675473 10646 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:48.676074 10646 heartbeater.cc:511] Master 127.0.97.62:35549 requested a full tablet report, sending...
I20260926 03:00:48.677420 10198 ts_manager.cc:194] Registered new tserver with Master: bb4f143bec704e5e9691116292426168 (127.0.97.3:36875)
I20260926 03:00:48.681922 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:00:48.687887 10387 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:00:48.686465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37576 (local address 127.0.97.62:35549)
0926 03:00:48.686624 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:48.686630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:48.687041 (+ 411us) server_negotiation.cc:407] Connection header received
0926 03:00:48.687220 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:48.687226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:48.687281 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:48.687378 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:48.687694 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37576: BlockingRecv error: recv got EOF from 127.0.0.1:37576 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:00:48.689118 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10258
I20260926 03:00:48.692231 10655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:48.690017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.0.97.1:34629)
0926 03:00:48.690481 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:48.690488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:48.690512 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:00:48.690563 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:48.690569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:48.690768 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:48.690880 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:48.691691 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:48.691697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:48.691709 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:48.691721 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:48.691761 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:48.691777 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:48.691872 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:48.691950 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 03:00:48.692026 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260926 03:00:48.693457 10380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:48.772360 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10258
I20260926 03:00:48.788414 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10389
I20260926 03:00:48.791896 10657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:48.789320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.0.97.2:43827)
0926 03:00:48.789786 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:48.789793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:48.789818 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:00:48.789876 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:48.789882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:48.790096 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:48.790266 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:48.791207 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:48.791212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:48.791224 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:48.791234 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:48.791271 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:48.791288 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:48.791428 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:48.791548 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 03:00:48.791665 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":214,"threads_started":1}
I20260926 03:00:48.793138 10511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:48.875571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10389
I20260926 03:00:48.891425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10519
I20260926 03:00:48.895428 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:48.892315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.0.97.3:36875)
0926 03:00:48.892936 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:48.892943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:48.892972 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:00:48.893086 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:48.893093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:48.893315 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:48.893480 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:48.894668 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:48.894673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:48.894685 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:00:48.894695 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:48.894745 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:00:48.894758 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:48.894934 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:48.895000 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 03:00:48.895104 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":323,"threads_started":1}
I20260926 03:00:48.896649 10641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:48.975530 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10519
I20260926 03:00:48.990908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10168
I20260926 03:00:48.992065 10229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:49.089787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10168
2026-09-26T03:00:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2244 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:00:49 dist-test-slave-fccw krb5kdc[10664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:00:49 dist-test-slave-fccw krb5kdc[10664](info): set up 2 sockets
Sep 26 03:00:49 dist-test-slave-fccw krb5kdc[10664](info): commencing operation
krb5kdc: starting...
W20260926 03:00:50.177917 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:00:50 dist-test-slave-fccw krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:00:50.393496 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37113
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:50.703228 10677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:50.703552 10677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:50.703720 10677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:50.710512 10677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:00:50.710637 10677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:50.710750 10677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:50.710878 10677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:00:50.710939 10677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:00:50.719988 10677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:50.721205 10677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:50.722435 10677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:50.733623 10682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:50.733599 10683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:50.733551 10685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:50.734499 10677 server_base.cc:1034] running on GCE node
I20260926 03:00:50.735250 10677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:50.736500 10677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:50.737686 10677 hybrid_clock.cc:648] HybridClock initialized: now 1790391650737645 us; error 57 us; skew 500 ppm
I20260926 03:00:50.737996 10677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:50 dist-test-slave-fccw krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:50.745711 10677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:00:50.747045 10677 webserver.cc:460] Webserver started at http://127.0.97.62:38767/ using document root <none> and password file <none>
I20260926 03:00:50.747634 10677 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:50.747725 10677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:50.747970 10677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:50.750748 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "f9e6601a881548d6ab97d4e62375cc18"
format_stamp: "Formatted at 2026-09-26 03:00:50 on dist-test-slave-fccw"
server_key: "d4055c6363020c2aa36ded49df114b0e"
server_key_iv: "ab7787ebbae3fbb76dc58e49fecffeca"
server_key_version: "encryptionkey@0"
I20260926 03:00:50.751415 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "f9e6601a881548d6ab97d4e62375cc18"
format_stamp: "Formatted at 2026-09-26 03:00:50 on dist-test-slave-fccw"
server_key: "d4055c6363020c2aa36ded49df114b0e"
server_key_iv: "ab7787ebbae3fbb76dc58e49fecffeca"
server_key_version: "encryptionkey@0"
I20260926 03:00:50.756089 10677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 03:00:50.759207 10692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:50.760390 10677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 03:00:50.760568 10677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "f9e6601a881548d6ab97d4e62375cc18"
format_stamp: "Formatted at 2026-09-26 03:00:50 on dist-test-slave-fccw"
server_key: "d4055c6363020c2aa36ded49df114b0e"
server_key_iv: "ab7787ebbae3fbb76dc58e49fecffeca"
server_key_version: "encryptionkey@0"
I20260926 03:00:50.760738 10677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:50.793071 10677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:50.797324 10677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:50.797636 10677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:50.815146 10677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37113
I20260926 03:00:50.815188 10743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37113 every 8 connection(s)
I20260926 03:00:50.816473 10677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:00:50.818311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10677
I20260926 03:00:50.818537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:00:50.819334 388 external_mini_cluster.cc:1428] Setting key fe2f7649492826008947c763f53b6124
I20260926 03:00:50.820909 10744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:00:50 dist-test-slave-fccw krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:50.834179 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18: Bootstrap starting.
I20260926 03:00:50.837201 10744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:50.838181 10744 log.cc:824] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:50.840864 10744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18: No bootstrap required, opened a new log
I20260926 03:00:50.840941 10747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:50.821368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.0.97.62:37113)
0926 03:00:50.822099 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:50.822110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:50.822146 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:00:50.823580 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:50.823598 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:50.823975 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:50.824282 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:50.826018 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:50.827310 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:50.829076 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:50.829685 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:50.829802 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:50.834322 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:50.834331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:50.834348 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:50.834393 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:50.837355 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:50.838015 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:50.838024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:50.838033 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:50.838186 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:50.838557 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:50.838561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:50.838563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:50.839045 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:50.839353 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:50.839713 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:00:50.840075 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":356,"threads_started":1}
I20260926 03:00:50.848313 10744 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6601a881548d6ab97d4e62375cc18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } }
I20260926 03:00:50.848657 10744 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:50.848765 10744 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e6601a881548d6ab97d4e62375cc18, State: Initialized, Role: FOLLOWER
I20260926 03:00:50.849407 10744 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6601a881548d6ab97d4e62375cc18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } }
I20260926 03:00:50.849694 10744 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:00:50.849848 10744 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:00:50.850037 10744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:50.851987 10744 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6601a881548d6ab97d4e62375cc18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } }
I20260926 03:00:50.852442 10744 leader_election.cc:302] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9e6601a881548d6ab97d4e62375cc18; no voters:
I20260926 03:00:50.853132 10744 leader_election.cc:288] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:00:50.853344 10749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:50.854089 10749 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [term 1 LEADER]: Becoming Leader. State: Replica: f9e6601a881548d6ab97d4e62375cc18, State: Running, Role: LEADER
I20260926 03:00:50.854704 10749 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6601a881548d6ab97d4e62375cc18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } }
I20260926 03:00:50.855240 10744 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:00:50.857630 10750 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9e6601a881548d6ab97d4e62375cc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6601a881548d6ab97d4e62375cc18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } }
I20260926 03:00:50.857983 10750 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:50.858503 10751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e6601a881548d6ab97d4e62375cc18. Latest consensus state: current_term: 1 leader_uuid: "f9e6601a881548d6ab97d4e62375cc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e6601a881548d6ab97d4e62375cc18" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37113 } } }
I20260926 03:00:50.858878 10751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18 [sys.catalog]: This master's current role is: LEADER
I20260926 03:00:50.864243 10756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:00:50.869493 10756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:00:50.877848 10756 catalog_manager.cc:1347] Generated new cluster ID: 0957b38b3eba4c5b97062b67368cb99b
I20260926 03:00:50.878015 10756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:00:50.888970 10756 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:00:50.890762 10756 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:00:50.901604 10756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18: Generated new TSK 0
I20260926 03:00:50.902356 10756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:00:50.966308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37113
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:51.293519 10770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:51.293785 10770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:51.293949 10770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:51.301056 10770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:51.301191 10770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:51.301379 10770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:00:51.310710 10770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:51.312011 10770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:51.313387 10770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:51.326172 10775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:51.326215 10776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:51.326261 10778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:51.327309 10770 server_base.cc:1034] running on GCE node
I20260926 03:00:51.327903 10770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:51.328895 10770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:51.330120 10770 hybrid_clock.cc:648] HybridClock initialized: now 1790391651330082 us; error 37 us; skew 500 ppm
I20260926 03:00:51.330435 10770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:51 dist-test-slave-fccw krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:51.333585 10770 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:00:51.335022 10770 webserver.cc:460] Webserver started at http://127.0.97.1:42435/ using document root <none> and password file <none>
I20260926 03:00:51.335711 10770 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:51.335855 10770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:51.336148 10770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:51.339032 10770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "282fdb22699e49a78ddbc79a2e78ec7f"
format_stamp: "Formatted at 2026-09-26 03:00:51 on dist-test-slave-fccw"
server_key: "559dd63ccfa3fd098998f72c9ae4f052"
server_key_iv: "7cc0660578b765b70fea91fee99ca10f"
server_key_version: "encryptionkey@0"
I20260926 03:00:51.339785 10770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "282fdb22699e49a78ddbc79a2e78ec7f"
format_stamp: "Formatted at 2026-09-26 03:00:51 on dist-test-slave-fccw"
server_key: "559dd63ccfa3fd098998f72c9ae4f052"
server_key_iv: "7cc0660578b765b70fea91fee99ca10f"
server_key_version: "encryptionkey@0"
I20260926 03:00:51.344689 10770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 03:00:51.348042 10785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:51.349346 10770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:00:51.349507 10770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "282fdb22699e49a78ddbc79a2e78ec7f"
format_stamp: "Formatted at 2026-09-26 03:00:51 on dist-test-slave-fccw"
server_key: "559dd63ccfa3fd098998f72c9ae4f052"
server_key_iv: "7cc0660578b765b70fea91fee99ca10f"
server_key_version: "encryptionkey@0"
I20260926 03:00:51.349704 10770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:51.389114 10770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:51.393057 10770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:51.393359 10770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:51.394304 10770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:51.395591 10770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:51.395691 10770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:51.395844 10770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:51.395927 10770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:51.429405 10770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40805
I20260926 03:00:51.429503 10897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40805 every 8 connection(s)
I20260926 03:00:51.430691 10770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:00:51.436767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10770
I20260926 03:00:51.436964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:00:51.437577 388 external_mini_cluster.cc:1428] Setting key 7fb7fc16e589d723a3b2dd06b0ceda78
Sep 26 03:00:51 dist-test-slave-fccw krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:51.450088 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:51.433099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36251 (local address 127.0.97.62:37113)
0926 03:00:51.433553 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:51.433557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:51.434712 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 03:00:51.436444 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:51.436450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:51.436559 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:51.436674 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:51.438279 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.439034 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.440509 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.440901 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.441071 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:51.444605 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:51.444610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:51.444613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:51.444641 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:51.447196 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:51.447884 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:51.447896 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:51.447899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:51.448010 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:51.448406 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:51.448411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:51.448415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:51.448660 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:51.448801 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:51.449536 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 03:00:51.449733 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":244,"threads_started":1}
I20260926 03:00:51.450989 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:51.433566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37113 (local address 127.0.97.1:36251)
0926 03:00:51.434385 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:51.434423 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:51.436187 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:51.436929 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:51.436938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:51.437487 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:51.438107 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:51.438124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.439178 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.439188 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:51.440362 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:51.440372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.441151 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.441159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:51.441392 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:51.442066 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:51.442086 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:51.444367 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:51.447340 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:51.447348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:51.447362 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:51.447755 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:51.448112 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:51.448118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:51.448121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:51.448293 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:51.448799 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:51.448807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:51.449176 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 03:00:51.449421 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 03:00:51.449907 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":257,"threads_started":1}
I20260926 03:00:51.452514 10898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37113
I20260926 03:00:51.452826 10898 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:51.453691 10898 heartbeater.cc:511] Master 127.0.97.62:37113 requested a full tablet report, sending...
I20260926 03:00:51.456080 10709 ts_manager.cc:194] Registered new tserver with Master: 282fdb22699e49a78ddbc79a2e78ec7f (127.0.97.1:40805)
I20260926 03:00:51.458247 10709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36251
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:00:51.487736 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37113
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:51.799325 10904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:51.799582 10904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:51.799674 10904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:51.806465 10904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:51.806559 10904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:51.806660 10904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:00:51.815868 10904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:51.817011 10904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:51.818284 10904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:51.830922 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:51.831012 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:51.831037 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:51.831457 10904 server_base.cc:1034] running on GCE node
I20260926 03:00:51.832052 10904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:51.832912 10904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:51.834100 10904 hybrid_clock.cc:648] HybridClock initialized: now 1790391651834071 us; error 60 us; skew 500 ppm
I20260926 03:00:51.834394 10904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:51 dist-test-slave-fccw krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:51.837390 10904 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:00:51.838680 10904 webserver.cc:460] Webserver started at http://127.0.97.2:36171/ using document root <none> and password file <none>
I20260926 03:00:51.839329 10904 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:51.839452 10904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:51.839743 10904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:51.842448 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "6e93531294b14641b3435716b2aa54ae"
format_stamp: "Formatted at 2026-09-26 03:00:51 on dist-test-slave-fccw"
server_key: "9dde88cdb0cc24fcea7120f8dca45b97"
server_key_iv: "083b52e0ea6e63eb34320a1059d06961"
server_key_version: "encryptionkey@0"
I20260926 03:00:51.843124 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6e93531294b14641b3435716b2aa54ae"
format_stamp: "Formatted at 2026-09-26 03:00:51 on dist-test-slave-fccw"
server_key: "9dde88cdb0cc24fcea7120f8dca45b97"
server_key_iv: "083b52e0ea6e63eb34320a1059d06961"
server_key_version: "encryptionkey@0"
I20260926 03:00:51.847854 10904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 03:00:51.850939 10919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:51.852155 10904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:00:51.852315 10904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "6e93531294b14641b3435716b2aa54ae"
format_stamp: "Formatted at 2026-09-26 03:00:51 on dist-test-slave-fccw"
server_key: "9dde88cdb0cc24fcea7120f8dca45b97"
server_key_iv: "083b52e0ea6e63eb34320a1059d06961"
server_key_version: "encryptionkey@0"
I20260926 03:00:51.852510 10904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:51.866328 10904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:51.870311 10904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:51.870589 10904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:51.871518 10904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:51.872727 10904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:51.872828 10904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:51.872973 10904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:51.873071 10904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 03:00:51.904857 10904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38397
I20260926 03:00:51.904898 11031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38397 every 8 connection(s)
I20260926 03:00:51.906124 10904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:00:51.907092 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 10904
I20260926 03:00:51.907286 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:00:51.907825 388 external_mini_cluster.cc:1428] Setting key b7f4a2e79ae60ed6c05b0ad2f68e71bd
Sep 26 03:00:51 dist-test-slave-fccw krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:51.927927 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:51.908970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38663 (local address 127.0.97.62:37113)
0926 03:00:51.909161 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:51.909165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:51.910487 (+ 1322us) server_negotiation.cc:407] Connection header received
0926 03:00:51.913344 (+ 2857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:51.913348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:51.913422 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:51.913514 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:51.915594 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.916270 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.918619 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.919037 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.919169 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:51.922562 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:51.922566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:51.922569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:51.922599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:51.924959 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:51.925657 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:51.925662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:51.925665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:51.925788 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:51.926220 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:51.926225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:51.926228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:51.926473 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:51.926623 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:51.927381 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 03:00:51.927595 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:00:51.928849 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:51.909466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37113 (local address 127.0.97.2:38663)
0926 03:00:51.910334 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:51.910381 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:51.913043 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:51.913786 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:51.913799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:51.914546 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:51.915325 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:51.915351 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.916418 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.916426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:51.918476 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:51.918487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:51.919253 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:51.919260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:51.919481 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:51.920159 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:51.920180 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:51.922342 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:51.925126 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:51.925135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:51.925150 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:51.925538 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:51.925889 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:51.925894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:51.925897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:51.926074 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:51.926601 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:51.926608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:51.926942 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 03:00:51.927289 (+ 347us) client_negotiation.cc:240] Negotiation successful
0926 03:00:51.927774 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":373,"threads_started":1}
I20260926 03:00:51.930392 11032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37113
I20260926 03:00:51.930680 11032 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:00:51.931384 11032 heartbeater.cc:511] Master 127.0.97.62:37113 requested a full tablet report, sending...
I20260926 03:00:51.932798 10709 ts_manager.cc:194] Registered new tserver with Master: 6e93531294b14641b3435716b2aa54ae (127.0.97.2:38397)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:00:51.933763 10709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:00:51.960021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37113
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:00:52.270716 11037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:00:52.270999 11037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:00:52.271087 11037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:00:52.277873 11037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:00:52.277971 11037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:00:52.278079 11037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:00:52.286942 11037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:00:52.288049 11037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:00:52.289233 11037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:00:52.302145 11043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:52.302238 11042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:00:52.302210 11045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:00:52.303207 11037 server_base.cc:1034] running on GCE node
I20260926 03:00:52.303686 11037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:00:52.304508 11037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:00:52.305694 11037 hybrid_clock.cc:648] HybridClock initialized: now 1790391652305663 us; error 30 us; skew 500 ppm
I20260926 03:00:52.305958 11037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:00:52 dist-test-slave-fccw krb5kdc[10664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:00:52.308981 11037 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:00:52.310289 11037 webserver.cc:460] Webserver started at http://127.0.97.3:34051/ using document root <none> and password file <none>
I20260926 03:00:52.310877 11037 fs_manager.cc:359] Metadata directory not provided
I20260926 03:00:52.310968 11037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:00:52.311214 11037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:00:52.314004 11037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "9186e648b4f8427a93e5560db9f2e6c3"
format_stamp: "Formatted at 2026-09-26 03:00:52 on dist-test-slave-fccw"
server_key: "51b7e8fcf8a6c1473ecb62d08bd8b23d"
server_key_iv: "140697574910415fa3a2097f1141fe4b"
server_key_version: "encryptionkey@0"
I20260926 03:00:52.314654 11037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9186e648b4f8427a93e5560db9f2e6c3"
format_stamp: "Formatted at 2026-09-26 03:00:52 on dist-test-slave-fccw"
server_key: "51b7e8fcf8a6c1473ecb62d08bd8b23d"
server_key_iv: "140697574910415fa3a2097f1141fe4b"
server_key_version: "encryptionkey@0"
I20260926 03:00:52.319279 11037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:00:52.322376 11052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:52.323565 11037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:00:52.323686 11037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "9186e648b4f8427a93e5560db9f2e6c3"
format_stamp: "Formatted at 2026-09-26 03:00:52 on dist-test-slave-fccw"
server_key: "51b7e8fcf8a6c1473ecb62d08bd8b23d"
server_key_iv: "140697574910415fa3a2097f1141fe4b"
server_key_version: "encryptionkey@0"
I20260926 03:00:52.323834 11037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:00:52.339998 11037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:00:52.343641 11037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:00:52.343874 11037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:00:52.344727 11037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:00:52.346051 11037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:00:52.346122 11037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:52.346205 11037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:00:52.346248 11037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:00:52.378015 11037 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40629
I20260926 03:00:52.378060 11164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40629 every 8 connection(s)
I20260926 03:00:52.379323 11037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:00:52.380138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 11037
I20260926 03:00:52.380324 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:00:52.380690 388 external_mini_cluster.cc:1428] Setting key 7b9dc2d6d28ceb6d14e148faa1f29817
Sep 26 03:00:52 dist-test-slave-fccw krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:00:52.398186 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.382982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43377 (local address 127.0.97.62:37113)
0926 03:00:52.383142 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.383148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.383430 (+ 282us) server_negotiation.cc:407] Connection header received
0926 03:00:52.385237 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.385241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.385301 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.385382 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:52.386732 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.387452 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.388886 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.389337 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.389428 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.392611 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:52.392615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:52.392618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:52.392647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:52.395351 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.396230 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.396235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.396238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.396425 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.396779 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.396782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.396784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.397088 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:52.397194 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.397805 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.397952 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:00:52.399127 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.382431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37113 (local address 127.0.97.3:43377)
0926 03:00:52.383239 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.383310 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.384950 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.385582 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.385590 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.386065 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:00:52.386586 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.386602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.387578 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.387582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.388771 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.388782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.389512 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.389517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.389709 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.390384 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:00:52.390409 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:00:52.392433 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:00:52.395499 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:52.395506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:52.395519 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:52.396106 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:52.396518 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:00:52.396522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:00:52.396524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:00:52.396686 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:00:52.397190 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:00:52.397196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:00:52.397489 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.397714 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.398064 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":283,"threads_started":1}
I20260926 03:00:52.400431 11165 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37113
I20260926 03:00:52.400728 11165 heartbeater.cc:464] Registering TS with master...
I20260926 03:00:52.401460 11165 heartbeater.cc:511] Master 127.0.97.62:37113 requested a full tablet report, sending...
I20260926 03:00:52.402767 10709 ts_manager.cc:194] Registered new tserver with Master: 9186e648b4f8427a93e5560db9f2e6c3 (127.0.97.3:40629)
I20260926 03:00:52.403479 10709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43377
I20260926 03:00:52.408160 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:00:52.423841 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.412460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.0.97.62:37113)
0926 03:00:52.412677 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.412686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.413031 (+ 345us) server_negotiation.cc:407] Connection header received
0926 03:00:52.413202 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.413205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.413261 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.413337 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:52.414703 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.415371 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.416479 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.416851 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.416937 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.419003 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:52.419008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:52.419011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:52.419039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:52.421664 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.422232 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.422236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.422238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.422342 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.422621 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.422624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.422626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.422905 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:52.423078 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.423357 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.423495 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:00:52.427549 10709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:00:52.434376 10709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:00:52.461313 10898 heartbeater.cc:503] Master 127.0.97.62:37113 was elected leader, sending a full tablet report...
I20260926 03:00:52.472090 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.445484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40805 (local address 127.0.0.1:39624)
0926 03:00:52.463064 (+ 17580us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.463080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.463260 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.464282 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.464290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.464322 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:52.464739 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.464758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.466808 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.466811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.468885 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.468897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.471061 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.471068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.471253 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.471297 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.471415 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.471619 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17444,"mutex_wait_us":1,"thread_start_us":17307,"threads_started":1}
I20260926 03:00:52.472680 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.446658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.0.97.3:40629)
0926 03:00:52.464693 (+ 18035us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.464704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.465510 (+ 806us) server_negotiation.cc:407] Connection header received
0926 03:00:52.465600 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.465607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.465824 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.465998 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:52.466760 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.468296 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.470406 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.471275 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.471427 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.471649 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.472186 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.472350 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17844,"thread_start_us":17676,"threads_started":1}
I20260926 03:00:52.473048 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.446008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.0.97.1:40805)
0926 03:00:52.456488 (+ 10480us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.456500 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.463166 (+ 6666us) server_negotiation.cc:407] Connection header received
0926 03:00:52.463438 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.463445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.463734 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.463947 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:52.464961 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.466669 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.469090 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.470583 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.470679 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.470890 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.472484 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.472661 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10261,"thread_start_us":10091,"threads_started":1}
I20260926 03:00:52.473153 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.445882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40629 (local address 127.0.0.1:55984)
0926 03:00:52.465260 (+ 19378us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.465280 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.465390 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.466239 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.466244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.466266 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:52.466558 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.466567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.468453 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.468462 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.470238 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.470249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.471496 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.471503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.471711 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.471742 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.471834 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.471958 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19238,"mutex_wait_us":1,"thread_start_us":18969,"threads_started":1}
I20260926 03:00:52.474206 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.445255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38397 (local address 127.0.0.1:60592)
0926 03:00:52.460430 (+ 15175us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.460490 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.460707 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.467236 (+ 6529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.467246 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.467270 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:52.467682 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.467693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.469533 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.469538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.471600 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.471612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.473426 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.473432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.473637 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.473669 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.473761 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.473877 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14830,"mutex_wait_us":1,"thread_start_us":15663,"threads_started":1}
I20260926 03:00:52.474296 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.445451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.97.2:38397)
0926 03:00:52.466628 (+ 21177us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.466635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.466683 (+ 48us) server_negotiation.cc:407] Connection header received
0926 03:00:52.466784 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.466792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.467017 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.467227 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:52.467847 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.469389 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.471741 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.473218 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.473344 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.473525 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.473838 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.473989 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21042,"thread_start_us":20917,"threads_started":1}
I20260926 03:00:52.477437 10967 tablet_service.cc:1467] Processing CreateTablet for tablet dcc08539fd734e39bbf7099721621403 (DEFAULT_TABLE table=test-table [id=d91910c129bc4c8aaa42230b34694114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:52.477476 10833 tablet_service.cc:1467] Processing CreateTablet for tablet dcc08539fd734e39bbf7099721621403 (DEFAULT_TABLE table=test-table [id=d91910c129bc4c8aaa42230b34694114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:52.477577 11100 tablet_service.cc:1467] Processing CreateTablet for tablet dcc08539fd734e39bbf7099721621403 (DEFAULT_TABLE table=test-table [id=d91910c129bc4c8aaa42230b34694114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:00:52.478606 10967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc08539fd734e39bbf7099721621403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:52.478768 10833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc08539fd734e39bbf7099721621403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:52.478785 11100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc08539fd734e39bbf7099721621403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:00:52.489387 11183 tablet_bootstrap.cc:492] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Bootstrap starting.
I20260926 03:00:52.492683 11185 tablet_bootstrap.cc:492] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Bootstrap starting.
I20260926 03:00:52.493043 11183 tablet_bootstrap.cc:654] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:52.493639 11184 tablet_bootstrap.cc:492] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Bootstrap starting.
I20260926 03:00:52.494664 11183 log.cc:824] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:52.496346 11185 tablet_bootstrap.cc:654] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:52.496884 11184 tablet_bootstrap.cc:654] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Neither blocks nor log segments found. Creating new log.
I20260926 03:00:52.497819 11184 log.cc:824] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:52.497824 11185 log.cc:824] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Log is configured to *not* fsync() on all Append() calls
I20260926 03:00:52.500535 11184 tablet_bootstrap.cc:492] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: No bootstrap required, opened a new log
I20260926 03:00:52.500785 11184 ts_tablet_manager.cc:1397] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 03:00:52.500895 11183 tablet_bootstrap.cc:492] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: No bootstrap required, opened a new log
I20260926 03:00:52.501055 11185 tablet_bootstrap.cc:492] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: No bootstrap required, opened a new log
I20260926 03:00:52.501189 11183 ts_tablet_manager.cc:1397] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260926 03:00:52.501299 11185 ts_tablet_manager.cc:1397] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 03:00:52.507885 11184 raft_consensus.cc:348] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.508235 11184 raft_consensus.cc:374] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:52.508347 11184 raft_consensus.cc:729] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282fdb22699e49a78ddbc79a2e78ec7f, State: Initialized, Role: FOLLOWER
I20260926 03:00:52.508917 11184 consensus_queue.cc:260] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.510702 11184 ts_tablet_manager.cc:1428] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260926 03:00:52.511716 11185 raft_consensus.cc:348] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.511716 11183 raft_consensus.cc:348] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.512154 11183 raft_consensus.cc:374] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:52.512167 11185 raft_consensus.cc:374] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:00:52.512260 11185 raft_consensus.cc:729] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9186e648b4f8427a93e5560db9f2e6c3, State: Initialized, Role: FOLLOWER
I20260926 03:00:52.512260 11183 raft_consensus.cc:729] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e93531294b14641b3435716b2aa54ae, State: Initialized, Role: FOLLOWER
I20260926 03:00:52.512879 11185 consensus_queue.cc:260] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.512859 11183 consensus_queue.cc:260] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.514842 11183 ts_tablet_manager.cc:1428] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 03:00:52.515017 11165 heartbeater.cc:503] Master 127.0.97.62:37113 was elected leader, sending a full tablet report...
I20260926 03:00:52.515262 11185 ts_tablet_manager.cc:1428] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 03:00:52.516333 11032 heartbeater.cc:503] Master 127.0.97.62:37113 was elected leader, sending a full tablet report...
I20260926 03:00:52.550426 11191 raft_consensus.cc:482] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:52.550681 11191 raft_consensus.cc:504] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.552213 11191 leader_election.cc:288] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e93531294b14641b3435716b2aa54ae (127.0.97.2:38397), 282fdb22699e49a78ddbc79a2e78ec7f (127.0.97.1:40805)
I20260926 03:00:52.558557 11190 raft_consensus.cc:482] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:00:52.558939 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.552756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47431 (local address 127.0.97.1:40805)
0926 03:00:52.552953 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.552959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.552980 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:00:52.553078 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.553086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.553179 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.553291 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:52.553839 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.554856 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.557033 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.558076 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.558202 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.558297 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.558576 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.558692 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:00:52.558966 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.552511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40805 (local address 127.0.97.3:47431)
0926 03:00:52.552737 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.552763 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.552891 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.553404 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.553416 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.553448 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:52.553690 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.553704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.554974 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.554978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.556853 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.556867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.558211 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.558237 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.558408 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.558444 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.558540 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.558669 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":24}
I20260926 03:00:52.558977 11190 raft_consensus.cc:504] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.560025 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.553076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54055 (local address 127.0.97.2:38397)
0926 03:00:52.553303 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.553310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.553803 (+ 493us) server_negotiation.cc:407] Connection header received
0926 03:00:52.553947 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.553954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.554041 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.554135 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:52.554599 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.555454 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.557541 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.558819 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.559013 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.559092 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.559624 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.559765 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:00:52.559981 10853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc08539fd734e39bbf7099721621403" candidate_uuid: "9186e648b4f8427a93e5560db9f2e6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" is_pre_election: true
I20260926 03:00:52.560559 10853 raft_consensus.cc:2454] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9186e648b4f8427a93e5560db9f2e6c3 in term 0.
I20260926 03:00:52.560793 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.552848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38397 (local address 127.0.97.3:54055)
0926 03:00:52.553674 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.553697 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.553818 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.554190 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.554194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.554222 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:52.554461 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.554468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.555598 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.555604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.557409 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.557418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.559224 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.559230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.559450 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.559480 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.560311 (+ 831us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.560426 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":2,"thread_start_us":469,"threads_started":1}
I20260926 03:00:52.561305 11053 leader_election.cc:302] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282fdb22699e49a78ddbc79a2e78ec7f, 9186e648b4f8427a93e5560db9f2e6c3; no voters:
I20260926 03:00:52.562259 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc08539fd734e39bbf7099721621403" candidate_uuid: "9186e648b4f8427a93e5560db9f2e6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e93531294b14641b3435716b2aa54ae" is_pre_election: true
I20260926 03:00:52.562633 11191 raft_consensus.cc:2781] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:00:52.562791 11191 raft_consensus.cc:482] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:00:52.562752 10987 raft_consensus.cc:2454] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9186e648b4f8427a93e5560db9f2e6c3 in term 0.
I20260926 03:00:52.562916 11191 raft_consensus.cc:3037] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:52.563949 11190 leader_election.cc:288] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 282fdb22699e49a78ddbc79a2e78ec7f (127.0.97.1:40805), 9186e648b4f8427a93e5560db9f2e6c3 (127.0.97.3:40629)
I20260926 03:00:52.566663 11191 raft_consensus.cc:504] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.567637 10987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc08539fd734e39bbf7099721621403" candidate_uuid: "9186e648b4f8427a93e5560db9f2e6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e93531294b14641b3435716b2aa54ae"
I20260926 03:00:52.567757 10853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc08539fd734e39bbf7099721621403" candidate_uuid: "9186e648b4f8427a93e5560db9f2e6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282fdb22699e49a78ddbc79a2e78ec7f"
I20260926 03:00:52.567821 10987 raft_consensus.cc:3037] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:52.567957 10853 raft_consensus.cc:3037] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:00:52.570151 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.561270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40805 (local address 127.0.97.2:56509)
0926 03:00:52.562558 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.562577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.562759 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.563254 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.563258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.563283 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:52.563675 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.563687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.565193 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.565215 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.567388 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.567401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.569308 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.569315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.569504 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.569541 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.569636 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.569765 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"thread_start_us":920,"threads_started":1}
I20260926 03:00:52.570741 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.561952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56509 (local address 127.0.97.1:40805)
0926 03:00:52.562505 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.562513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.562659 (+ 146us) server_negotiation.cc:407] Connection header received
0926 03:00:52.562932 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.562936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.563006 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.563069 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:52.563825 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.565058 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.567557 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.568884 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.569028 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.569147 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.570356 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.570480 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432}
I20260926 03:00:52.570828 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.561536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45849 (local address 127.0.97.3:40629)
0926 03:00:52.561713 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.561719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.564593 (+ 2874us) server_negotiation.cc:407] Connection header received
0926 03:00:52.564735 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.564740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.564812 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.564899 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:00:52.565620 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.566312 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.568512 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.569720 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.569811 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.569913 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.570476 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.570599 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:00:52.571108 10987 raft_consensus.cc:2454] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9186e648b4f8427a93e5560db9f2e6c3 in term 1.
I20260926 03:00:52.571200 10852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc08539fd734e39bbf7099721621403" candidate_uuid: "6e93531294b14641b3435716b2aa54ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" is_pre_election: true
I20260926 03:00:52.571441 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.561439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40629 (local address 127.0.97.2:45849)
0926 03:00:52.564480 (+ 3041us) negotiation.cc:107] Waiting for socket to connect
0926 03:00:52.564497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:00:52.564612 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:00:52.564977 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:00:52.564983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:00:52.565028 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:00:52.565442 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.565456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.566444 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.566449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.568333 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:00:52.568348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.570121 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.570133 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:00:52.570304 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.570336 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:00:52.570976 (+ 640us) client_negotiation.cc:240] Negotiation successful
0926 03:00:52.571121 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2929,"thread_start_us":2769,"threads_started":1}
I20260926 03:00:52.571532 10853 raft_consensus.cc:2454] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9186e648b4f8427a93e5560db9f2e6c3 in term 1.
I20260926 03:00:52.571918 11053 leader_election.cc:302] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282fdb22699e49a78ddbc79a2e78ec7f, 9186e648b4f8427a93e5560db9f2e6c3; no voters:
I20260926 03:00:52.572134 11191 leader_election.cc:288] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [CANDIDATE]: Term 1 election: Requested vote from peers 6e93531294b14641b3435716b2aa54ae (127.0.97.2:38397), 282fdb22699e49a78ddbc79a2e78ec7f (127.0.97.1:40805)
I20260926 03:00:52.572340 11191 raft_consensus.cc:2781] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:00:52.572309 11120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc08539fd734e39bbf7099721621403" candidate_uuid: "6e93531294b14641b3435716b2aa54ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9186e648b4f8427a93e5560db9f2e6c3" is_pre_election: true
I20260926 03:00:52.572755 11191 raft_consensus.cc:686] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 1 LEADER]: Becoming Leader. State: Replica: 9186e648b4f8427a93e5560db9f2e6c3, State: Running, Role: LEADER
I20260926 03:00:52.573334 11191 consensus_queue.cc:237] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } }
I20260926 03:00:52.573685 10922 leader_election.cc:302] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6e93531294b14641b3435716b2aa54ae; no voters: 282fdb22699e49a78ddbc79a2e78ec7f, 9186e648b4f8427a93e5560db9f2e6c3
I20260926 03:00:52.574039 11190 raft_consensus.cc:2726] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 03:00:52.577610 10709 catalog_manager.cc:5495] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9186e648b4f8427a93e5560db9f2e6c3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9186e648b4f8427a93e5560db9f2e6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9186e648b4f8427a93e5560db9f2e6c3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40629 } health_report { overall_health: HEALTHY } } }
I20260926 03:00:52.596338 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.591604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.0.97.3:40629)
0926 03:00:52.591767 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.591772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.591980 (+ 208us) server_negotiation.cc:407] Connection header received
0926 03:00:52.592132 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.592135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.592220 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.592330 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:00:52.592617 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.593410 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.594562 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.594914 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.594996 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.595431 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:00:52.595896 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:00:52.595999 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.596057 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.596112 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:00:52.603895 10853 raft_consensus.cc:1261] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 1 FOLLOWER]: Refusing update from remote peer 9186e648b4f8427a93e5560db9f2e6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:52.603941 10987 raft_consensus.cc:1261] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 1 FOLLOWER]: Refusing update from remote peer 9186e648b4f8427a93e5560db9f2e6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:00:52.604725 11195 consensus_queue.cc:1035] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282fdb22699e49a78ddbc79a2e78ec7f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:52.605065 11191 consensus_queue.cc:1035] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e93531294b14641b3435716b2aa54ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:00:52.620898 11198 mvcc.cc:205] Tried to move back new op lower bound from 7333444209056276480 to 7333444208944652288. Current Snapshot: MvccSnapshot[applied={T|T < 7333444209056276480}]
I20260926 03:00:52.621335 11200 mvcc.cc:205] Tried to move back new op lower bound from 7333444209056276480 to 7333444208944652288. Current Snapshot: MvccSnapshot[applied={T|T < 7333444209056276480}]
I20260926 03:00:52.636297 10708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:52.636569 10708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:00:52.640053 10708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9e6601a881548d6ab97d4e62375cc18: Sending DeleteTablet for 3 replicas of tablet dcc08539fd734e39bbf7099721621403
I20260926 03:00:52.641094 10967 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc08539fd734e39bbf7099721621403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60592
I20260926 03:00:52.641129 11100 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc08539fd734e39bbf7099721621403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55984
I20260926 03:00:52.641158 10833 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc08539fd734e39bbf7099721621403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:00:52 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39624
I20260926 03:00:52.644047 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10770
I20260926 03:00:52.651881 11209 tablet_replica.cc:330] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: stopping tablet replica
I20260926 03:00:52.653141 11209 raft_consensus.cc:2229] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:52.654435 11209 raft_consensus.cc:2258] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:52.655812 11207 tablet_replica.cc:330] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: stopping tablet replica
I20260926 03:00:52.656353 11207 raft_consensus.cc:2229] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:00:52.657156 11207 raft_consensus.cc:2258] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:52.658587 11209 ts_tablet_manager.cc:1905] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:52.659260 11207 ts_tablet_manager.cc:1905] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:52.659780 11208 tablet_replica.cc:330] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: stopping tablet replica
Sep 26 03:00:52 dist-test-slave-fccw krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:00:52.663679 11208 raft_consensus.cc:2229] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:00:52.664201 11208 raft_consensus.cc:2258] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:00:52.666087 11208 ts_tablet_manager.cc:1905] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:00:52.666106 11209 ts_tablet_manager.cc:1918] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:52.666258 11209 log.cc:1196] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/dcc08539fd734e39bbf7099721621403
I20260926 03:00:52.666862 11209 ts_tablet_manager.cc:1939] T dcc08539fd734e39bbf7099721621403 P 282fdb22699e49a78ddbc79a2e78ec7f: Deleting consensus metadata
I20260926 03:00:52.668603 10693 catalog_manager.cc:4841] TS 282fdb22699e49a78ddbc79a2e78ec7f (127.0.97.1:40805): tablet dcc08539fd734e39bbf7099721621403 (table test-table [id=d91910c129bc4c8aaa42230b34694114]) successfully deleted
I20260926 03:00:52.670434 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.645432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.0.97.1:40805)
0926 03:00:52.645592 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.645598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.651302 (+ 5704us) server_negotiation.cc:407] Connection header received
0926 03:00:52.651506 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.651513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.651607 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.651723 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:52.653488 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.654609 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.656840 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.657590 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.657693 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.662810 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:52.662823 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:52.662841 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:52.662880 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:52.667069 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.667955 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.667963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.667966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.668113 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.668560 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.668565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.668570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.668987 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:52.669313 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.669878 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.670085 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:00:52.671980 10893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:52.672581 11207 ts_tablet_manager.cc:1918] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:52.672616 11208 ts_tablet_manager.cc:1918] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:00:52.672736 11208 log.cc:1196] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/dcc08539fd734e39bbf7099721621403
I20260926 03:00:52.672783 11207 log.cc:1196] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/dcc08539fd734e39bbf7099721621403
I20260926 03:00:52.673288 11208 ts_tablet_manager.cc:1939] T dcc08539fd734e39bbf7099721621403 P 6e93531294b14641b3435716b2aa54ae: Deleting consensus metadata
I20260926 03:00:52.673313 11207 ts_tablet_manager.cc:1939] T dcc08539fd734e39bbf7099721621403 P 9186e648b4f8427a93e5560db9f2e6c3: Deleting consensus metadata
I20260926 03:00:52.674788 10694 catalog_manager.cc:4841] TS 6e93531294b14641b3435716b2aa54ae (127.0.97.2:38397): tablet dcc08539fd734e39bbf7099721621403 (table test-table [id=d91910c129bc4c8aaa42230b34694114]) successfully deleted
I20260926 03:00:52.675182 10695 catalog_manager.cc:4841] TS 9186e648b4f8427a93e5560db9f2e6c3 (127.0.97.3:40629): tablet dcc08539fd734e39bbf7099721621403 (table test-table [id=d91910c129bc4c8aaa42230b34694114]) successfully deleted
I20260926 03:00:52.761844 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10770
I20260926 03:00:52.779137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10904
Sep 26 03:00:52 dist-test-slave-fccw krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:00:52.794920 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.779923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.97.2:38397)
0926 03:00:52.780102 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.780109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.780128 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:00:52.780332 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.780344 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.780405 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.780511 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:52.781741 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.782427 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.783628 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.784005 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.784126 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.788116 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:52.788133 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:52.788154 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:52.788199 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:52.791818 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.792556 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.792566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.792570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.792734 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.793307 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.793312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.793316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.793797 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:52.794074 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.794310 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.794537 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:00:52.795675 11027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:52.884543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10904
I20260926 03:00:52.901575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 11037
Sep 26 03:00:52 dist-test-slave-fccw krb5kdc[10664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:00:52.918253 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:00:52.902383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.0.97.3:40629)
0926 03:00:52.902580 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:00:52.902586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:00:52.902607 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:00:52.902753 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:00:52.902758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:00:52.902857 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:00:52.902972 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:00:52.904146 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.905296 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.906495 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:00:52.907215 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:00:52.907354 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:00:52.911547 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:00:52.911558 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:00:52.911577 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:00:52.911636 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:00:52.915392 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.915989 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.915997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.916001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.916152 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:00:52.916463 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:00:52.916468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:00:52.916471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:00:52.916933 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:00:52.917261 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 03:00:52.917687 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 03:00:52.917896 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 03:00:52.919013 11160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:53.036556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 11037
I20260926 03:00:53.057300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 10677
I20260926 03:00:53.058313 10739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:00:53.167923 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 10677
2026-09-26T03:00:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4079 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:00:53 dist-test-slave-fccw krb5kdc[11217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:00:53 dist-test-slave-fccw krb5kdc[11217](info): set up 2 sockets
Sep 26 03:00:53 dist-test-slave-fccw krb5kdc[11217](info): commencing operation
krb5kdc: starting...
W20260926 03:00:54.262275 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:00:54 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:00:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 03:00:54.561269 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task33M2KT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/postgres -l logfile start
2026-09-26 03:00:56.962 UTC [11247] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:00:56.962 UTC [11247] LOG: listening on IPv4 address "127.0.97.20", port 37381
2026-09-26 03:00:56.965 UTC [11247] LOG: listening on Unix socket "/tmp/.s.PGSQL.37381"
2026-09-26 03:00:56.988 UTC [11250] LOG: database system was shut down at 2026-09-26 03:00:55 UTC
2026-09-26 03:00:56.992 UTC [11247] LOG: database system is ready to accept connections
I20260926 03:00:57.957520 388 mini_postgres.cc:96] Postgres bound to 37381
2026-09-26 03:00:57.974 UTC [11258] FATAL: database "slave" does not exist
127.0.97.20:37381 - accepting connections
I20260926 03:00:57.974846 388 mini_ranger.cc:161] Starting Ranger...
I20260926 03:00:57.999089 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:00:58.449460 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:00:58.449575 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-admin
2026-09-26 03:00:58,724 [I] DB FLAVOR :POSTGRES
2026-09-26 03:00:58,725 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:00:58,725 [I] Checking connection..
2026-09-26 03:00:58,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:00:58,975 [I] Checking connection passed.
2026-09-26 03:00:58,976 [I] --------- Verifying version history table ---------
2026-09-26 03:00:58,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:00:59,219 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:00:59,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:00:59,467 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:00:59,468 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:00:59,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:00:59.717 UTC [11322] WARNING: there is no transaction in progress
2026-09-26 03:00:59,723 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:00:59,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:00:59,956 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:00:59,957 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:00:59,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:01:00,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:01:00,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fccw.c.gcp-upstream.internal','N') ;"
2026-09-26 03:01:00,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:01:00,939 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:01:00,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:01:01,193 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:01:01,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:01:01,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:01:01,645 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:01:01,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:01:02.571 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.587 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.602 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.617 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.631 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.646 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.694 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.700 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.713 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.722 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.731 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.743 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.752 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.757 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.764 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:02.769 UTC [11430] WARNING: there is no transaction in progress
2026-09-26 03:01:03,360 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:01:03,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:01:03,582 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:01:03,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:01:03,806 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:01:03,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:01:04,041 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:01:04,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:01:04,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:01:04,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fccw.c.gcp-upstream.internal';"
2026-09-26 03:01:04,745 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:01:04,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:01:04,988 [I] DB_PATCHES have already been applied
I20260926 03:01:04.994132 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task33M2KT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:01:04.994217 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 03:01:05.008843 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34817
Sep 26, 2026 3:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:01:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:01:06.131431 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.157s user 0.009s sys 0.026s
Sep 26, 2026 3:01:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:01:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 3:01:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:01:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:01:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:01:26.334419 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:01:26 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391686, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:01:26.487242 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:01:26.505292 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:01:27.344326 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:01:27.344491 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms
I20260926 03:01:27.344624 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:37381
2026-09-26 03:01:27,753 [I] DB FLAVOR :POSTGRES
2026-09-26 03:01:27,754 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:01:27,754 [I] Checking connection
2026-09-26 03:01:27,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:01:27,996 [I] connection success
2026-09-26 03:01:27,997 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:01:27,997 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:01:27,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:01:28,311 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:01:28,311 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:01:28,311 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:01:28,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:01:28,609 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:01:29.062305 388 mini_ranger_kms.cc:310] Created kms service
I20260926 03:01:29.267925 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:01:29.311537 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:01:29 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391686, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:01:29.711767 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:01:29.712091 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:01:29.712167 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 03:01:29.728808 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11610
I20260926 03:01:29.729033 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:41809
Sep 26, 2026 3:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:01:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:01:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:01:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 03:01:31 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391691, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:01:31.935175 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.448s user 0.137s sys 0.283s
I20260926 03:01:31.935421 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:01:31.935638 388 mini_ranger_kms.cc:401] 127.0.97.20:41809/kms/v1/keys
Sep 26 03:01:32 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391691, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:01:32 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391691, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:01:34 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:01:34.804746 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44087
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44087
--ranger_config_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41809/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:01:35.153162 11661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:01:35.153417 11661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:01:35.153589 11661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:01:35.160343 11661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:01:35.160434 11661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:01:35.160477 11661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:01:35.160549 11661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:01:35.160633 11661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:01:35.170776 11661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:41809/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44087
--ranger_config_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:01:35.171960 11661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:01:35.173310 11661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:01:35.185093 11666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:35.185046 11667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:35.185137 11669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:01:35.185645 11661 server_base.cc:1034] running on GCE node
I20260926 03:01:35.186318 11661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:01:35.187536 11661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:01:35.188766 11661 hybrid_clock.cc:648] HybridClock initialized: now 1790391695188718 us; error 38 us; skew 500 ppm
I20260926 03:01:35.189199 11661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:01:35 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:01:35.192534 11661 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:01:35.193914 11661 webserver.cc:460] Webserver started at http://127.0.97.62:38803/ using document root <none> and password file <none>
I20260926 03:01:35.194569 11661 fs_manager.cc:359] Metadata directory not provided
I20260926 03:01:35.194720 11661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:01:35.194990 11661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:01:35 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:01:35.243701 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "772625fe9ebb422e95b0858698abce80"
format_stamp: "Formatted at 2026-09-26 03:01:35 on dist-test-slave-fccw"
server_key: "f3d800d3dec287d8c90094aef068a2c4"
server_key_iv: "70c951a1f5bb3801e8b05036e328bb2f"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:35.244517 11661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "772625fe9ebb422e95b0858698abce80"
format_stamp: "Formatted at 2026-09-26 03:01:35 on dist-test-slave-fccw"
server_key: "f3d800d3dec287d8c90094aef068a2c4"
server_key_iv: "70c951a1f5bb3801e8b05036e328bb2f"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:35.249967 11661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:01:35.268725 11676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:01:35.270292 11661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:01:35.270476 11661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "772625fe9ebb422e95b0858698abce80"
format_stamp: "Formatted at 2026-09-26 03:01:35 on dist-test-slave-fccw"
server_key: "f3d800d3dec287d8c90094aef068a2c4"
server_key_iv: "70c951a1f5bb3801e8b05036e328bb2f"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:35.270697 11661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:01:35.291029 11661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:01:35.294809 11661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:01:35.295099 11661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:01:35.316764 11661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44087
I20260926 03:01:35.316793 11727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44087 every 8 connection(s)
I20260926 03:01:35.318102 11661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:01:35.321619 11728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:01:35.323004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 11661
I20260926 03:01:35.323191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:01:35.323807 388 external_mini_cluster.cc:1428] Setting key d9f22af9f4e8adf2e32abe84da4288ee
I20260926 03:01:35.332652 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80: Bootstrap starting.
I20260926 03:01:35.336233 11728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80: Neither blocks nor log segments found. Creating new log.
I20260926 03:01:35.337500 11728 log.cc:824] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:01:35 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:01:35.340124 11728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80: No bootstrap required, opened a new log
I20260926 03:01:35.345968 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:35.325656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.0.97.62:44087)
0926 03:01:35.326411 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:35.326421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:35.326461 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:01:35.328058 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:35.328077 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:35.328434 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:35.328721 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:35.330216 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:35.331872 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:35.333639 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:35.334295 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:35.334436 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:35.338840 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:35.338848 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:35.338862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:35.338913 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:35.341951 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:35.342645 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:35.342651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:35.342657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:35.342804 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:35.343210 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:35.343214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:35.343216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:35.343672 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:35.343973 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:35.344482 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 03:01:35.344854 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":377,"threads_started":1}
I20260926 03:01:35.348215 11728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772625fe9ebb422e95b0858698abce80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44087 } }
I20260926 03:01:35.348584 11728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:01:35.348711 11728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772625fe9ebb422e95b0858698abce80, State: Initialized, Role: FOLLOWER
I20260926 03:01:35.349331 11728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772625fe9ebb422e95b0858698abce80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44087 } }
I20260926 03:01:35.349571 11728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:01:35.349709 11728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:01:35.349880 11728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:01:35.351989 11728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772625fe9ebb422e95b0858698abce80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44087 } }
I20260926 03:01:35.352473 11728 leader_election.cc:302] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 772625fe9ebb422e95b0858698abce80; no voters:
I20260926 03:01:35.353096 11728 leader_election.cc:288] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:01:35.353241 11733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:01:35.353993 11733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [term 1 LEADER]: Becoming Leader. State: Replica: 772625fe9ebb422e95b0858698abce80, State: Running, Role: LEADER
I20260926 03:01:35.354517 11733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772625fe9ebb422e95b0858698abce80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44087 } }
I20260926 03:01:35.355126 11728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:01:35.357265 11734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "772625fe9ebb422e95b0858698abce80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772625fe9ebb422e95b0858698abce80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44087 } } }
I20260926 03:01:35.358526 11734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [sys.catalog]: This master's current role is: LEADER
I20260926 03:01:35.357409 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 772625fe9ebb422e95b0858698abce80. Latest consensus state: current_term: 1 leader_uuid: "772625fe9ebb422e95b0858698abce80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "772625fe9ebb422e95b0858698abce80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44087 } } }
I20260926 03:01:35.358915 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80 [sys.catalog]: This master's current role is: LEADER
I20260926 03:01:35.362892 11728 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:01:35.364530 11742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:01:35.375255 11742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:01:35.387446 11742 catalog_manager.cc:1347] Generated new cluster ID: ee0c2aba99864ee7817a9c2008997d02
I20260926 03:01:35.387569 11742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:01:35.439410 11742 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:01:35.441386 11742 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:01:35.452230 11742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80: Generated new TSK 0
I20260926 03:01:35.453958 11742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:01:36.121338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44087
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:01:36 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:01:36.543819 11773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:01:36.544212 11773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:01:36.544406 11773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:01:36.555702 11773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:01:36.555845 11773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:01:36.556087 11773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:01:36.571818 11773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:41809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:01:36.573529 11773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:01:36.575263 11773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:01:36.604105 11783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:36.609822 11784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:36.610774 11786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:01:36.614681 11773 server_base.cc:1034] running on GCE node
I20260926 03:01:36.615523 11773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:01:36.616796 11773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:01:36.618078 11773 hybrid_clock.cc:648] HybridClock initialized: now 1790391696618014 us; error 73 us; skew 500 ppm
I20260926 03:01:36.618486 11773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:01:36 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:01:36.621925 11773 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:01:36.626169 11773 webserver.cc:460] Webserver started at http://127.0.97.1:33371/ using document root <none> and password file <none>
I20260926 03:01:36.626878 11773 fs_manager.cc:359] Metadata directory not provided
I20260926 03:01:36.626999 11773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:01:36.627362 11773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:01:36 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:01:36.644884 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "acf2fdee8d4c4df3a2575ee9bd778926"
format_stamp: "Formatted at 2026-09-26 03:01:36 on dist-test-slave-fccw"
server_key: "a5bf87a9b1528830058afcbf24bb9cbc"
server_key_iv: "ac480456d2490975b4d57948bd8b968d"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:36.646049 11773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "acf2fdee8d4c4df3a2575ee9bd778926"
format_stamp: "Formatted at 2026-09-26 03:01:36 on dist-test-slave-fccw"
server_key: "a5bf87a9b1528830058afcbf24bb9cbc"
server_key_iv: "ac480456d2490975b4d57948bd8b968d"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:36.660723 11773 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.003s sys 0.002s
I20260926 03:01:36.685235 11793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:01:36.687350 11773 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260926 03:01:36.687618 11773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "acf2fdee8d4c4df3a2575ee9bd778926"
format_stamp: "Formatted at 2026-09-26 03:01:36 on dist-test-slave-fccw"
server_key: "a5bf87a9b1528830058afcbf24bb9cbc"
server_key_iv: "ac480456d2490975b4d57948bd8b968d"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:36.687966 11773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:01:36.739323 11773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:01:36.744573 11773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:01:36.744967 11773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:01:36.746773 11773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:01:36.748670 11773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:01:36.748808 11773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:01:36.749001 11773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:01:36.749166 11773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:01:36 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:01:36.871855 11905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36747 every 8 connection(s)
I20260926 03:01:36.872664 11773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36747
I20260926 03:01:36.874540 11773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:01:36.882532 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 11773
I20260926 03:01:36.882709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:01:36.883374 388 external_mini_cluster.cc:1428] Setting key 8f95ad839b78a21a2fa0d6950e91b696
Sep 26 03:01:36 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:01:36.930135 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:36.884768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44087 (local address 127.0.97.1:46627)
0926 03:01:36.896686 (+ 11918us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:36.896722 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:36.899076 (+ 2354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:36.900193 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:36.900206 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:36.900798 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:01:36.901716 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:36.901742 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:36.903842 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:36.903847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:36.911241 (+ 7394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:36.911260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:36.912579 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:36.912585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:36.912873 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:36.917429 (+ 4556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:01:36.917461 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:01:36.920260 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:01:36.924597 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:01:36.924608 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:01:36.924631 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:01:36.925296 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:01:36.926190 (+ 894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:01:36.926195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:01:36.926199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:01:36.926459 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:01:36.927197 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:01:36.927206 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:01:36.927749 (+ 543us) client_negotiation.cc:769] Sending connection context
0926 03:01:36.928078 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 03:01:36.928670 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11666,"thread_start_us":11923,"threads_started":1}
I20260926 03:01:36.930646 11909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:36.885535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46627 (local address 127.0.97.62:44087)
0926 03:01:36.895386 (+ 9851us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:36.895390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:36.896844 (+ 1454us) server_negotiation.cc:407] Connection header received
0926 03:01:36.899519 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:36.899524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:36.899617 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:36.899722 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:36.901926 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:36.903121 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:36.911430 (+ 8309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:36.912131 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:36.912233 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:36.920561 (+ 8328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:36.920569 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:36.920574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:36.920620 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:36.924400 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:36.925467 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:36.925475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:36.925480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:36.926019 (+ 539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:36.926623 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:36.926629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:36.926636 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:36.927020 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:36.927908 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:36.928200 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 03:01:36.928433 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9739,"thread_start_us":9639,"threads_started":1}
I20260926 03:01:36.932287 11906 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44087
I20260926 03:01:36.932622 11906 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:01:36.933548 11906 heartbeater.cc:511] Master 127.0.97.62:44087 requested a full tablet report, sending...
I20260926 03:01:36.935902 11693 ts_manager.cc:194] Registered new tserver with Master: acf2fdee8d4c4df3a2575ee9bd778926 (127.0.97.1:36747)
I20260926 03:01:36.937927 11693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:01:36.972922 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44087
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:01:37 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:01:37.411901 11912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:01:37.412423 11912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:01:37.412621 11912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:01:37.424659 11912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:01:37.424875 11912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:01:37.425127 11912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:01:37.440698 11912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:41809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:01:37.442629 11912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:01:37.444834 11912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:01:37.460754 11919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:37.466020 11920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:37.474470 11922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:01:37.474874 11912 server_base.cc:1034] running on GCE node
I20260926 03:01:37.475654 11912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:01:37.476861 11912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:01:37.478211 11912 hybrid_clock.cc:648] HybridClock initialized: now 1790391697478129 us; error 69 us; skew 500 ppm
I20260926 03:01:37.478621 11912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:01:37 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:01:37.482650 11912 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:01:37.500466 11912 webserver.cc:460] Webserver started at http://127.0.97.2:36891/ using document root <none> and password file <none>
I20260926 03:01:37.501540 11912 fs_manager.cc:359] Metadata directory not provided
I20260926 03:01:37.501691 11912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:01:37.502051 11912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:01:37 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:01:37.520725 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "aba14b8fa07c43f8a7912bdb50a11e66"
format_stamp: "Formatted at 2026-09-26 03:01:37 on dist-test-slave-fccw"
server_key: "8f47d21e8c50d907bb7c8ce7613dfdd6"
server_key_iv: "1b8caef3c5f5b4444b4abb8d11ba794b"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:37.521746 11912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "aba14b8fa07c43f8a7912bdb50a11e66"
format_stamp: "Formatted at 2026-09-26 03:01:37 on dist-test-slave-fccw"
server_key: "8f47d21e8c50d907bb7c8ce7613dfdd6"
server_key_iv: "1b8caef3c5f5b4444b4abb8d11ba794b"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:37.530726 11912 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260926 03:01:37.560179 11936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:01:37.562568 11912 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.009s
I20260926 03:01:37.562835 11912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "aba14b8fa07c43f8a7912bdb50a11e66"
format_stamp: "Formatted at 2026-09-26 03:01:37 on dist-test-slave-fccw"
server_key: "8f47d21e8c50d907bb7c8ce7613dfdd6"
server_key_iv: "1b8caef3c5f5b4444b4abb8d11ba794b"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:37.563185 11912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:01:37.604604 11912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:01:37.609998 11912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:01:37.610424 11912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:01:37.611740 11912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:01:37.613538 11912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:01:37.613670 11912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 03:01:37.613819 11912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:01:37.613915 11912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:01:37.661113 11912 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38871
I20260926 03:01:37.661206 12048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38871 every 8 connection(s)
I20260926 03:01:37.662429 11912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:01:37.664795 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 11912
I20260926 03:01:37.665541 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:01:37.666250 388 external_mini_cluster.cc:1428] Setting key a56df834a67af32d9156a6cd4b17d7fc
Sep 26 03:01:37 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:01:37.695710 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:37.665320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55103 (local address 127.0.97.62:44087)
0926 03:01:37.678531 (+ 13211us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:37.678539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:37.678563 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:01:37.679983 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:37.679992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:37.680096 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:37.680214 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:37.682067 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:37.683321 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:37.684844 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:37.685588 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:37.685733 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:37.689269 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:37.689277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:37.689282 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:37.689325 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:37.692548 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:37.693318 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:37.693323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:37.693326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:37.693473 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:37.693931 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:37.693934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:37.693936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:37.694325 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:37.694493 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:37.695291 (+ 798us) server_negotiation.cc:299] Negotiation successful
0926 03:01:37.695452 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13074,"thread_start_us":12962,"threads_started":1}
I20260926 03:01:37.697261 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:37.665791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44087 (local address 127.0.97.2:55103)
0926 03:01:37.677768 (+ 11977us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:37.677836 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:37.679715 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:37.680428 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:37.680436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:37.680871 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:01:37.681898 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:37.681916 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:37.683469 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:37.683473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:37.684696 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:37.684707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:37.685786 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:37.685790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:37.685969 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:37.686638 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:01:37.686660 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:01:37.689061 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:01:37.692668 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:01:37.692674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:01:37.692687 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:01:37.693220 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:01:37.693629 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:01:37.693633 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:01:37.693635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:01:37.693803 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:01:37.694439 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:01:37.694445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:01:37.694892 (+ 447us) client_negotiation.cc:769] Sending connection context
0926 03:01:37.695203 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 03:01:37.695783 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11569,"thread_start_us":11315,"threads_started":1}
I20260926 03:01:37.698745 12049 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44087
I20260926 03:01:37.699041 12049 heartbeater.cc:464] Registering TS with master...
I20260926 03:01:37.699707 12049 heartbeater.cc:511] Master 127.0.97.62:44087 requested a full tablet report, sending...
I20260926 03:01:37.701237 11693 ts_manager.cc:194] Registered new tserver with Master: aba14b8fa07c43f8a7912bdb50a11e66 (127.0.97.2:38871)
I20260926 03:01:37.702338 11693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:01:37.723793 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:41809/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44087
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:01:37.945887 11906 heartbeater.cc:503] Master 127.0.97.62:44087 was elected leader, sending a full tablet report...
W20260926 03:01:38.047139 12055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:01:38.047442 12055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:01:38.047605 12055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:01:38.054945 12055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:01:38.055080 12055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:01:38.055267 12055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:01:38.064666 12055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:41809/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:01:38.065939 12055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:01:38.067185 12055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:01:38.080351 12061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:38.080437 12063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:01:38.080457 12060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:01:38.081352 12055 server_base.cc:1034] running on GCE node
I20260926 03:01:38.081907 12055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:01:38.082801 12055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:01:38.084024 12055 hybrid_clock.cc:648] HybridClock initialized: now 1790391698083973 us; error 38 us; skew 500 ppm
I20260926 03:01:38.084342 12055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:01:38 dist-test-slave-fccw krb5kdc[11217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:01:38.087355 12055 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:01:38.088668 12055 webserver.cc:460] Webserver started at http://127.0.97.3:39935/ using document root <none> and password file <none>
I20260926 03:01:38.089411 12055 fs_manager.cc:359] Metadata directory not provided
I20260926 03:01:38.089548 12055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:01:38.089856 12055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:01:38 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:01:38.105749 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "f5b35cb26c1a442184323136d7e9d777"
format_stamp: "Formatted at 2026-09-26 03:01:38 on dist-test-slave-fccw"
server_key: "29195b3bba9be50792a3c904f342f630"
server_key_iv: "44ea7b61071133a9a5fbd2701d4aed5a"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:38.106775 12055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f5b35cb26c1a442184323136d7e9d777"
format_stamp: "Formatted at 2026-09-26 03:01:38 on dist-test-slave-fccw"
server_key: "29195b3bba9be50792a3c904f342f630"
server_key_iv: "44ea7b61071133a9a5fbd2701d4aed5a"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:38.113168 12055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 03:01:38.128188 12070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:01:38.129801 12055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:01:38.129968 12055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "f5b35cb26c1a442184323136d7e9d777"
format_stamp: "Formatted at 2026-09-26 03:01:38 on dist-test-slave-fccw"
server_key: "29195b3bba9be50792a3c904f342f630"
server_key_iv: "44ea7b61071133a9a5fbd2701d4aed5a"
server_key_version: "kuduclusterkey@0"
I20260926 03:01:38.130165 12055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:01:38.146401 12055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:01:38.150213 12055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:01:38.150492 12055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:01:38.151480 12055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:01:38.152792 12055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:01:38.152863 12055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:01:38.153049 12055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:01:38.153136 12055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 03:01:38.188720 12055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43545
I20260926 03:01:38.188751 12182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43545 every 8 connection(s)
I20260926 03:01:38.189996 12055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:01:38.195451 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 12055
I20260926 03:01:38.195628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:01:38.196070 388 external_mini_cluster.cc:1428] Setting key 0333711190b1cf2db889e32ed968dc1a
Sep 26 03:01:38 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:01:38.209430 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.192765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49849 (local address 127.0.97.62:44087)
0926 03:01:38.192926 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.192931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.194243 (+ 1312us) server_negotiation.cc:407] Connection header received
0926 03:01:38.196062 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.196068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.196198 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.196344 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:38.198119 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.198866 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.200409 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.200842 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.200976 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.204034 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:38.204039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:38.204042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:38.204071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:38.206720 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.207407 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.207410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.207413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.207532 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.207897 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.207900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.207902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.208195 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:38.208300 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.208858 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.209091 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:01:38.210533 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.193299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44087 (local address 127.0.97.3:49849)
0926 03:01:38.194091 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:38.194136 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:38.195771 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:38.196591 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:38.196602 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:38.197166 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:01:38.197954 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.197974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.199050 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.199055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.200261 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.200273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.201073 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.201081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.201321 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.201907 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:01:38.201928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:01:38.203844 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:01:38.206852 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:01:38.206858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:01:38.206869 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:01:38.207301 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:01:38.207622 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:01:38.207625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:01:38.207627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:01:38.207801 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:01:38.208304 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:01:38.208310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:01:38.208566 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 03:01:38.208774 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 03:01:38.209172 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":362,"threads_started":1}
I20260926 03:01:38.211848 12183 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44087
I20260926 03:01:38.212138 12183 heartbeater.cc:464] Registering TS with master...
I20260926 03:01:38.212790 12183 heartbeater.cc:511] Master 127.0.97.62:44087 requested a full tablet report, sending...
I20260926 03:01:38.214531 11693 ts_manager.cc:194] Registered new tserver with Master: f5b35cb26c1a442184323136d7e9d777 (127.0.97.3:43545)
I20260926 03:01:38.215353 11693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49849
I20260926 03:01:38.223330 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:01:38.239691 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.227701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.97.62:44087)
0926 03:01:38.227913 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.227923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.228284 (+ 361us) server_negotiation.cc:407] Connection header received
0926 03:01:38.228403 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.228407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.228460 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.228537 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:38.229909 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.230624 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.231972 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.232394 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.232489 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.234833 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:38.234838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:38.234841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:38.234869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:38.237293 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.237857 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.237860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.237862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.237974 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.238270 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.238273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.238275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.238501 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:38.239123 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.239278 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.239453 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:01:38.243270 11693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:01:38.249868 11693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:01:38.282117 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.263183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.0.97.3:43545)
0926 03:01:38.270202 (+ 7019us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.270215 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.270265 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:01:38.270382 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.270393 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.270752 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.271893 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:01:38.273598 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.276324 (+ 2726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.279495 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.280866 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.281019 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.281265 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.281568 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.281750 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6803,"thread_start_us":6613,"threads_started":1}
I20260926 03:01:38.282137 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.261344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36747 (local address 127.0.0.1:51220)
0926 03:01:38.263662 (+ 2318us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:38.263708 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:38.263899 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:38.274929 (+ 11030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:38.274933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:38.274956 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:01:38.275382 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.275399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.277283 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.277288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.279602 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.279614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.281387 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.281392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.281561 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.281593 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:01:38.281663 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 03:01:38.281781 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1933,"thread_start_us":7675,"threads_started":1}
I20260926 03:01:38.282194 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.262227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43545 (local address 127.0.0.1:41340)
0926 03:01:38.269368 (+ 7141us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:38.269391 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:38.269530 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:38.271042 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:38.271053 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:38.271095 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:01:38.271633 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.271649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.276474 (+ 4825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.276483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.279335 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.279347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.281071 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.281077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.281286 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.281365 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 03:01:38.281509 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 03:01:38.281696 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6971,"mutex_wait_us":1,"thread_start_us":1239,"threads_started":1}
I20260926 03:01:38.283279 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.270192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38871 (local address 127.0.0.1:57878)
0926 03:01:38.275668 (+ 5476us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:38.275687 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:38.275825 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:38.276368 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:38.276373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:38.276397 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:01:38.276824 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.276835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.279184 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.279190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.280667 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.280678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.282541 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.282547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.282709 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.282740 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:01:38.282849 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:01:38.282964 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5373,"thread_start_us":5205,"threads_started":1}
I20260926 03:01:38.283362 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.269700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.0.97.2:38871)
0926 03:01:38.274009 (+ 4309us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.274018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.275810 (+ 1792us) server_negotiation.cc:407] Connection header received
0926 03:01:38.275972 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.275980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.276176 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.276365 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:01:38.277126 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.279037 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.280857 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.282331 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.282487 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.282676 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.282896 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.283049 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4127,"thread_start_us":4799,"threads_started":1}
I20260926 03:01:38.283401 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.261631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.0.97.1:36747)
0926 03:01:38.274459 (+ 12828us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.274465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.274487 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:01:38.274565 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.274570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.274752 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.274953 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:01:38.275614 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.277122 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.279759 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.281147 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.282614 (+ 1467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.282835 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.282913 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.283087 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12671,"thread_start_us":12525,"threads_started":1}
I20260926 03:01:38.286791 11841 tablet_service.cc:1467] Processing CreateTablet for tablet 40fc136fa8754cca8a91dc57a80e4ad4 (DEFAULT_TABLE table=test-table [id=563309a02cad4b56af712d6f9b02bd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:01:38.286791 12118 tablet_service.cc:1467] Processing CreateTablet for tablet 40fc136fa8754cca8a91dc57a80e4ad4 (DEFAULT_TABLE table=test-table [id=563309a02cad4b56af712d6f9b02bd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:01:38.287559 11984 tablet_service.cc:1467] Processing CreateTablet for tablet 40fc136fa8754cca8a91dc57a80e4ad4 (DEFAULT_TABLE table=test-table [id=563309a02cad4b56af712d6f9b02bd32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:01:38.288285 11841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fc136fa8754cca8a91dc57a80e4ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:01:38.288275 12118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fc136fa8754cca8a91dc57a80e4ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:01:38.288693 11984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40fc136fa8754cca8a91dc57a80e4ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:01:38.301106 12201 tablet_bootstrap.cc:492] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Bootstrap starting.
I20260926 03:01:38.304471 12201 tablet_bootstrap.cc:654] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Neither blocks nor log segments found. Creating new log.
I20260926 03:01:38.305526 12201 log.cc:824] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Log is configured to *not* fsync() on all Append() calls
I20260926 03:01:38.306854 12202 tablet_bootstrap.cc:492] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Bootstrap starting.
I20260926 03:01:38.308687 12203 tablet_bootstrap.cc:492] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Bootstrap starting.
I20260926 03:01:38.309128 12201 tablet_bootstrap.cc:492] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: No bootstrap required, opened a new log
I20260926 03:01:38.309399 12201 ts_tablet_manager.cc:1397] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260926 03:01:38.310734 12202 tablet_bootstrap.cc:654] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Neither blocks nor log segments found. Creating new log.
I20260926 03:01:38.311862 12202 log.cc:824] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Log is configured to *not* fsync() on all Append() calls
I20260926 03:01:38.312983 12203 tablet_bootstrap.cc:654] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Neither blocks nor log segments found. Creating new log.
I20260926 03:01:38.314332 12203 log.cc:824] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Log is configured to *not* fsync() on all Append() calls
I20260926 03:01:38.314733 12202 tablet_bootstrap.cc:492] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: No bootstrap required, opened a new log
I20260926 03:01:38.314976 12202 ts_tablet_manager.cc:1397] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 03:01:38.319908 12201 raft_consensus.cc:348] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.320392 12203 tablet_bootstrap.cc:492] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: No bootstrap required, opened a new log
I20260926 03:01:38.320667 12203 ts_tablet_manager.cc:1397] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260926 03:01:38.320338 12201 raft_consensus.cc:374] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:01:38.320837 12201 raft_consensus.cc:729] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba14b8fa07c43f8a7912bdb50a11e66, State: Initialized, Role: FOLLOWER
I20260926 03:01:38.321485 12201 consensus_queue.cc:260] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.327225 12202 raft_consensus.cc:348] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.327777 12202 raft_consensus.cc:374] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:01:38.327896 12202 raft_consensus.cc:729] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b35cb26c1a442184323136d7e9d777, State: Initialized, Role: FOLLOWER
I20260926 03:01:38.328722 12202 consensus_queue.cc:260] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.329974 12049 heartbeater.cc:503] Master 127.0.97.62:44087 was elected leader, sending a full tablet report...
I20260926 03:01:38.330680 12201 ts_tablet_manager.cc:1428] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260926 03:01:38.330765 12203 raft_consensus.cc:348] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.331140 12203 raft_consensus.cc:374] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:01:38.331235 12203 raft_consensus.cc:729] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acf2fdee8d4c4df3a2575ee9bd778926, State: Initialized, Role: FOLLOWER
I20260926 03:01:38.331784 12203 consensus_queue.cc:260] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.334590 12183 heartbeater.cc:503] Master 127.0.97.62:44087 was elected leader, sending a full tablet report...
I20260926 03:01:38.335572 12202 ts_tablet_manager.cc:1428] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260926 03:01:38.341339 12203 ts_tablet_manager.cc:1428] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Time spent starting tablet: real 0.021s user 0.013s sys 0.004s
W20260926 03:01:38.389325 11907 tablet.cc:2369] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:01:38.397321 12208 raft_consensus.cc:482] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:01:38.397564 12208 raft_consensus.cc:504] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.399082 12208 leader_election.cc:288] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acf2fdee8d4c4df3a2575ee9bd778926 (127.0.97.1:36747), aba14b8fa07c43f8a7912bdb50a11e66 (127.0.97.2:38871)
I20260926 03:01:38.404928 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.399528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36747 (local address 127.0.97.3:59585)
0926 03:01:38.399722 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:38.399743 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:38.399907 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:38.400558 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:38.400562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:38.400591 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:01:38.400774 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.400782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.401835 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.401841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.403135 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.403146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.404226 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.404231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.404411 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.404435 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:01:38.404541 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:01:38.404666 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":23}
I20260926 03:01:38.404927 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.399517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59585 (local address 127.0.97.1:36747)
0926 03:01:38.399721 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.399728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.399830 (+ 102us) server_negotiation.cc:407] Connection header received
0926 03:01:38.400341 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.400347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.400430 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.400535 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:01:38.400907 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.401696 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.403292 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.404074 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.404219 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.404318 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.404567 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.404684 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:01:38.405895 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc136fa8754cca8a91dc57a80e4ad4" candidate_uuid: "f5b35cb26c1a442184323136d7e9d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" is_pre_election: true
I20260926 03:01:38.406368 11861 raft_consensus.cc:2454] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b35cb26c1a442184323136d7e9d777 in term 0.
I20260926 03:01:38.406726 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.400172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49373 (local address 127.0.97.2:38871)
0926 03:01:38.400352 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.400357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.400792 (+ 435us) server_negotiation.cc:407] Connection header received
0926 03:01:38.401038 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.401042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.401102 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.401203 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:01:38.401756 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.402495 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.404692 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.405739 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.405906 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.406015 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.406312 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.406452 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:01:38.406823 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.399977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38871 (local address 127.0.97.3:49373)
0926 03:01:38.400677 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 03:01:38.400692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:01:38.400867 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:01:38.401214 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:01:38.401218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:01:38.401255 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:01:38.401580 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.401589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.402649 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.402654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.404092 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:01:38.404104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.405956 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.405962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:01:38.406135 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.406194 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 03:01:38.406287 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 03:01:38.406417 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":1,"thread_start_us":394,"threads_started":1}
I20260926 03:01:38.407195 12072 leader_election.cc:302] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acf2fdee8d4c4df3a2575ee9bd778926, f5b35cb26c1a442184323136d7e9d777; no voters:
I20260926 03:01:38.407552 12208 raft_consensus.cc:2781] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:01:38.407707 12208 raft_consensus.cc:482] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:01:38.407809 12208 raft_consensus.cc:3037] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:01:38.408306 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc136fa8754cca8a91dc57a80e4ad4" candidate_uuid: "f5b35cb26c1a442184323136d7e9d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" is_pre_election: true
I20260926 03:01:38.408720 12004 raft_consensus.cc:2454] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5b35cb26c1a442184323136d7e9d777 in term 0.
I20260926 03:01:38.410467 12208 raft_consensus.cc:504] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.411090 12208 leader_election.cc:288] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [CANDIDATE]: Term 1 election: Requested vote from peers acf2fdee8d4c4df3a2575ee9bd778926 (127.0.97.1:36747), aba14b8fa07c43f8a7912bdb50a11e66 (127.0.97.2:38871)
I20260926 03:01:38.411515 11861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc136fa8754cca8a91dc57a80e4ad4" candidate_uuid: "f5b35cb26c1a442184323136d7e9d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acf2fdee8d4c4df3a2575ee9bd778926"
I20260926 03:01:38.411538 12004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40fc136fa8754cca8a91dc57a80e4ad4" candidate_uuid: "f5b35cb26c1a442184323136d7e9d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba14b8fa07c43f8a7912bdb50a11e66"
I20260926 03:01:38.411700 11861 raft_consensus.cc:3037] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:01:38.411738 12004 raft_consensus.cc:3037] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:01:38.414260 11861 raft_consensus.cc:2454] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b35cb26c1a442184323136d7e9d777 in term 1.
I20260926 03:01:38.414623 12072 leader_election.cc:302] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acf2fdee8d4c4df3a2575ee9bd778926, f5b35cb26c1a442184323136d7e9d777; no voters:
I20260926 03:01:38.414772 12004 raft_consensus.cc:2454] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5b35cb26c1a442184323136d7e9d777 in term 1.
I20260926 03:01:38.414888 12208 raft_consensus.cc:2781] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:01:38.415464 12208 raft_consensus.cc:686] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 1 LEADER]: Becoming Leader. State: Replica: f5b35cb26c1a442184323136d7e9d777, State: Running, Role: LEADER
I20260926 03:01:38.415957 12208 consensus_queue.cc:237] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } }
I20260926 03:01:38.420703 11693 catalog_manager.cc:5495] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5b35cb26c1a442184323136d7e9d777 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f5b35cb26c1a442184323136d7e9d777" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b35cb26c1a442184323136d7e9d777" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43545 } health_report { overall_health: HEALTHY } } }
W20260926 03:01:38.424203 12050 tablet.cc:2369] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:01:38.441869 12184 tablet.cc:2369] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:01:38.462209 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.457087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.0.97.3:43545)
0926 03:01:38.457250 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.457255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.457286 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:01:38.457489 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.457492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.457571 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.457691 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:01:38.458023 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.458708 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.460232 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.460680 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.460766 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.461289 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:01:38.461765 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:01:38.461869 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.461966 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.462026 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:01:38.469791 12004 raft_consensus.cc:1261] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 1 FOLLOWER]: Refusing update from remote peer f5b35cb26c1a442184323136d7e9d777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:01:38.469842 11861 raft_consensus.cc:1261] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 1 FOLLOWER]: Refusing update from remote peer f5b35cb26c1a442184323136d7e9d777: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:01:38.470705 12211 consensus_queue.cc:1035] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aba14b8fa07c43f8a7912bdb50a11e66" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:01:38.471062 12208 consensus_queue.cc:1035] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acf2fdee8d4c4df3a2575ee9bd778926" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:01:38.498015 11691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:01:38.498256 11691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:01:38.501370 11691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 772625fe9ebb422e95b0858698abce80: Sending DeleteTablet for 3 replicas of tablet 40fc136fa8754cca8a91dc57a80e4ad4
I20260926 03:01:38.502506 11984 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fc136fa8754cca8a91dc57a80e4ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:01:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57878
I20260926 03:01:38.502511 12118 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fc136fa8754cca8a91dc57a80e4ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:01:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41340
I20260926 03:01:38.503533 11841 tablet_service.cc:1514] Processing DeleteTablet for tablet 40fc136fa8754cca8a91dc57a80e4ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:01:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51220
I20260926 03:01:38.504647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 11773
I20260926 03:01:38.515110 12224 tablet_replica.cc:330] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: stopping tablet replica
I20260926 03:01:38.515733 12223 tablet_replica.cc:330] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: stopping tablet replica
I20260926 03:01:38.516685 12225 tablet_replica.cc:330] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: stopping tablet replica
I20260926 03:01:38.516902 12224 raft_consensus.cc:2229] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:01:38.516278 12223 raft_consensus.cc:2229] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:01:38.517226 12225 raft_consensus.cc:2229] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:01:38.517710 12225 raft_consensus.cc:2258] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:01:38.517710 12224 raft_consensus.cc:2258] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:01:38.518271 12223 raft_consensus.cc:2258] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:01:38.520008 12225 ts_tablet_manager.cc:1905] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:01:38.520476 12223 ts_tablet_manager.cc:1905] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:01:38 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:01:38.526942 12225 ts_tablet_manager.cc:1918] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:01:38.527082 12225 log.cc:1196] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/40fc136fa8754cca8a91dc57a80e4ad4
I20260926 03:01:38.527566 12225 ts_tablet_manager.cc:1939] T 40fc136fa8754cca8a91dc57a80e4ad4 P acf2fdee8d4c4df3a2575ee9bd778926: Deleting consensus metadata
I20260926 03:01:38.527621 12223 ts_tablet_manager.cc:1918] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:01:38.527791 12223 log.cc:1196] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/40fc136fa8754cca8a91dc57a80e4ad4
I20260926 03:01:38.528280 12223 ts_tablet_manager.cc:1939] T 40fc136fa8754cca8a91dc57a80e4ad4 P f5b35cb26c1a442184323136d7e9d777: Deleting consensus metadata
I20260926 03:01:38.529917 11678 catalog_manager.cc:4841] TS acf2fdee8d4c4df3a2575ee9bd778926 (127.0.97.1:36747): tablet 40fc136fa8754cca8a91dc57a80e4ad4 (table test-table [id=563309a02cad4b56af712d6f9b02bd32]) successfully deleted
I20260926 03:01:38.530789 12224 ts_tablet_manager.cc:1905] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:01:38.530853 11679 catalog_manager.cc:4841] TS f5b35cb26c1a442184323136d7e9d777 (127.0.97.3:43545): tablet 40fc136fa8754cca8a91dc57a80e4ad4 (table test-table [id=563309a02cad4b56af712d6f9b02bd32]) successfully deleted
I20260926 03:01:38.535624 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.507585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.0.97.1:36747)
0926 03:01:38.507759 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.507765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.516040 (+ 8275us) server_negotiation.cc:407] Connection header received
0926 03:01:38.516217 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.516225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.516317 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.516445 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:38.518369 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.519531 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.521565 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.522237 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.522344 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.528236 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:38.528252 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:38.528270 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:38.528315 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:38.532279 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.533172 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.533181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.533185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.533349 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.533854 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.533859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.533862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.534278 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:38.534603 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.534976 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.535218 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:01:38.536445 11901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:01:38.536830 12224 ts_tablet_manager.cc:1918] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:01:38.537024 12224 log.cc:1196] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/40fc136fa8754cca8a91dc57a80e4ad4
I20260926 03:01:38.537537 12224 ts_tablet_manager.cc:1939] T 40fc136fa8754cca8a91dc57a80e4ad4 P aba14b8fa07c43f8a7912bdb50a11e66: Deleting consensus metadata
I20260926 03:01:38.539276 11678 catalog_manager.cc:4841] TS aba14b8fa07c43f8a7912bdb50a11e66 (127.0.97.2:38871): tablet 40fc136fa8754cca8a91dc57a80e4ad4 (table test-table [id=563309a02cad4b56af712d6f9b02bd32]) successfully deleted
I20260926 03:01:38.626271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 11773
I20260926 03:01:38.644754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 11912
Sep 26 03:01:38 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:01:38.661646 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.645695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.0.97.2:38871)
0926 03:01:38.645886 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.645893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.645915 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:01:38.646008 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.646014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.646093 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.646206 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:38.647413 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.648520 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.649814 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.650453 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.650568 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.654404 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:38.654421 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:38.654441 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:38.654480 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:38.658449 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.659242 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.659251 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.659255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.659399 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.659893 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.659917 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.659921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.660425 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:38.660768 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.661040 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.661247 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:01:38.662382 12044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:01:38.750465 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 11912
I20260926 03:01:38.772395 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 12055
Sep 26 03:01:38 dist-test-slave-fccw krb5kdc[11217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:01:38.790218 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:01:38.773202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.0.97.3:43545)
0926 03:01:38.773397 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:01:38.773402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:01:38.773424 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:01:38.773567 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:01:38.773577 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:01:38.773666 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:01:38.773788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:01:38.775184 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.776413 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.777929 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:01:38.778599 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:01:38.778742 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:01:38.783301 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:01:38.783316 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:01:38.783335 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:01:38.783408 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:01:38.787183 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.787884 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.787900 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.787904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.788081 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:01:38.788441 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:01:38.788445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:01:38.788448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:01:38.788903 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:01:38.789461 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0926 03:01:38.789633 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 03:01:38.789838 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:01:38.790941 12178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:01:38.913749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 12055
I20260926 03:01:38.932487 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 11661
I20260926 03:01:38.933738 11723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:01:39.049412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 11661
I20260926 03:01:39.071962 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:01:39.406487 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:01:39.406705 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:01:39.750869 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:01:39.751 UTC [11247] LOG: received smart shutdown request
2026-09-26 03:01:39.754 UTC [11247] LOG: background worker "logical replication launcher" (PID 11256) exited with exit code 1
2026-09-26 03:01:39.754 UTC [11251] LOG: shutting down
2026-09-26 03:01:39.912 UTC [11247] LOG: database system is shut down
2026-09-26T03:01:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46860 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:01:40 dist-test-slave-fccw krb5kdc[12239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:01:40 dist-test-slave-fccw krb5kdc[12239](info): set up 2 sockets
Sep 26 03:01:40 dist-test-slave-fccw krb5kdc[12239](info): commencing operation
krb5kdc: starting...
W20260926 03:01:41.129397 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:01:41 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:01:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260926 03:01:41.441879 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task33M2KT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/postgres -l logfile start
2026-09-26 03:01:43.839 UTC [12269] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:01:43.840 UTC [12269] LOG: listening on IPv4 address "127.0.97.20", port 33355
2026-09-26 03:01:43.843 UTC [12269] LOG: listening on Unix socket "/tmp/.s.PGSQL.33355"
2026-09-26 03:01:43.864 UTC [12272] LOG: database system was shut down at 2026-09-26 03:01:42 UTC
2026-09-26 03:01:43.867 UTC [12269] LOG: database system is ready to accept connections
I20260926 03:01:44.833261 388 mini_postgres.cc:96] Postgres bound to 33355
2026-09-26 03:01:44.849 UTC [12280] FATAL: database "slave" does not exist
127.0.97.20:33355 - accepting connections
I20260926 03:01:44.850474 388 mini_ranger.cc:161] Starting Ranger...
I20260926 03:01:44.875082 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:01:45.337241 388 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:01:45.337354 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-admin
2026-09-26 03:01:45,619 [I] DB FLAVOR :POSTGRES
2026-09-26 03:01:45,619 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:01:45,619 [I] Checking connection..
2026-09-26 03:01:45,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:01:45,874 [I] Checking connection passed.
2026-09-26 03:01:45,874 [I] --------- Verifying version history table ---------
2026-09-26 03:01:45,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:01:46,124 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:01:46,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:01:46,361 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:01:46,362 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:01:46,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:01:46.588 UTC [12344] WARNING: there is no transaction in progress
2026-09-26 03:01:46,595 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:01:46,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:01:46,848 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:01:46,848 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:01:46,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:01:47,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:01:47,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fccw.c.gcp-upstream.internal','N') ;"
2026-09-26 03:01:47,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:01:47,829 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:01:47,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:01:48,084 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:01:48,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:01:48,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:01:48,562 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:01:48,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:01:49.482 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.498 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.513 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.528 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.542 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.557 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.608 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.614 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.627 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.636 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.646 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.658 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.667 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.673 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.679 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:49.685 UTC [12452] WARNING: there is no transaction in progress
2026-09-26 03:01:50,267 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:01:50,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:01:50,487 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:01:50,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:01:50,741 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:01:50,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:01:50,987 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:01:50,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:01:51,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:01:51,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fccw.c.gcp-upstream.internal';"
2026-09-26 03:01:51,719 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:01:51,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:01:51,954 [I] DB_PATCHES have already been applied
I20260926 03:01:51.960361 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task33M2KT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:01:51.960619 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260926 03:01:51.974414 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:41053
Sep 26, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:01:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:01:53.084947 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.234s user 0.005s sys 0.031s
Sep 26, 2026 3:02:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:02:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 3:02:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:02:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:02:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:02:13.742745 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:02:13 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391733, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:02:13.902040 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:02:13.922415 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:02:14.729858 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:02:14.729982 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms
I20260926 03:02:14.730033 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:33355
2026-09-26 03:02:15,067 [I] DB FLAVOR :POSTGRES
2026-09-26 03:02:15,067 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:02:15,067 [I] Checking connection
2026-09-26 03:02:15,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:02:15,333 [I] connection success
2026-09-26 03:02:15,334 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:02:15,334 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:02:15,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:02:15,592 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:02:15,592 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:02:15,592 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:02:15,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33355/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:02:15,882 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:02:16.349459 388 mini_ranger_kms.cc:310] Created kms service
I20260926 03:02:16.532387 388 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:02:16.575784 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:02:16 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391733, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:16.939458 388 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:02:16.939793 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task33M2KT/build/asan/bin/postgresql.jar:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task33M2KT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:02:16.939858 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260926 03:02:16.958238 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12632
I20260926 03:02:16.958370 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:50525
Sep 26, 2026 3:02:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:02:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:02:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:02:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:02:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:02:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:02:19.229173 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.327s user 0.054s sys 0.296s
I20260926 03:02:19.229424 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:02:19.229588 388 mini_ranger_kms.cc:401] 127.0.97.20:50525/kms/v1/keys
Sep 26 03:02:19 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:02:19 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:02:20 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:02:22 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:02:22.416532 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46109
--ranger_config_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:22.729909 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:22.730226 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:22.730367 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:22.737115 12683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:22.737262 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:22.737344 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:22.737423 12683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:22.737496 12683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:22.746800 12683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:50525/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46109
--ranger_config_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:22.747965 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:22.749235 12683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:22.761175 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:22.761248 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:22.761200 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:22.761574 12683 server_base.cc:1034] running on GCE node
I20260926 03:02:22.762456 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:22.763749 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:22.764984 12683 hybrid_clock.cc:648] HybridClock initialized: now 1790391742764946 us; error 33 us; skew 500 ppm
I20260926 03:02:22.765388 12683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:22 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:22.776472 12683 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:02:22.778021 12683 webserver.cc:460] Webserver started at http://127.0.97.62:42853/ using document root <none> and password file <none>
I20260926 03:02:22.778684 12683 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:22.778828 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:22.779137 12683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:02:22 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:22.837280 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "ff367b148cce49d9b7f9bf434523fcb6"
format_stamp: "Formatted at 2026-09-26 03:02:22 on dist-test-slave-fccw"
server_key: "cf72ddb4ef36bf275593a4599653efa1"
server_key_iv: "c8d7eeec0fff791949390670287b77e5"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:22.838104 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff367b148cce49d9b7f9bf434523fcb6"
format_stamp: "Formatted at 2026-09-26 03:02:22 on dist-test-slave-fccw"
server_key: "cf72ddb4ef36bf275593a4599653efa1"
server_key_iv: "c8d7eeec0fff791949390670287b77e5"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:22.843269 12683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 03:02:22.861819 12698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:22.863337 12683 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:02:22.863512 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "ff367b148cce49d9b7f9bf434523fcb6"
format_stamp: "Formatted at 2026-09-26 03:02:22 on dist-test-slave-fccw"
server_key: "cf72ddb4ef36bf275593a4599653efa1"
server_key_iv: "c8d7eeec0fff791949390670287b77e5"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:22.863716 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:22.874629 12683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:22.878540 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:22.878866 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:22.898836 12683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46109
I20260926 03:02:22.898856 12749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46109 every 8 connection(s)
I20260926 03:02:22.900163 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:02:22.904063 12750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:22.907361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 12683
I20260926 03:02:22.907590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:02:22.908435 388 external_mini_cluster.cc:1428] Setting key e558f79ec51c950d7fb98e73bc79c58b
I20260926 03:02:22.919938 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Bootstrap starting.
I20260926 03:02:22.923861 12750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:22.924974 12750 log.cc:824] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:22.928177 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: No bootstrap required, opened a new log
Sep 26 03:02:22 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:22.937443 12750 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:22.937959 12750 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:22.938084 12750 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff367b148cce49d9b7f9bf434523fcb6, State: Initialized, Role: FOLLOWER
I20260926 03:02:22.938788 12750 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:22.939087 12750 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:02:22.939248 12750 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:02:22.939429 12750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:22.942487 12750 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:22.943063 12750 leader_election.cc:302] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff367b148cce49d9b7f9bf434523fcb6; no voters:
I20260926 03:02:22.943724 12750 leader_election.cc:288] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:02:22.943843 12755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:02:22.944013 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:22.910583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.97.62:46109)
0926 03:02:22.921995 (+ 11412us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:22.922014 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:22.922068 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:02:22.924622 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:22.924652 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:22.925189 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:22.926364 (+ 1175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:22.927232 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:22.929203 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:22.930545 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:22.931664 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:22.931869 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:22.935762 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:22.935773 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:22.935798 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:22.935851 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:22.939621 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:22.940269 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:22.940281 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:22.940290 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:22.940467 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:22.940822 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:22.940827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:22.940830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:22.941574 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:22.941874 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:22.942166 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 03:02:22.942592 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11146,"thread_start_us":10903,"threads_started":1}
I20260926 03:02:22.945228 12755 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 LEADER]: Becoming Leader. State: Replica: ff367b148cce49d9b7f9bf434523fcb6, State: Running, Role: LEADER
I20260926 03:02:22.945784 12755 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:22.947409 12750 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:22.953691 12750 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:02:22.954062 12756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff367b148cce49d9b7f9bf434523fcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260926 03:02:22.954473 12756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:22.954094 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff367b148cce49d9b7f9bf434523fcb6. Latest consensus state: current_term: 1 leader_uuid: "ff367b148cce49d9b7f9bf434523fcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260926 03:02:22.955196 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:22.957862 12770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:02:22.969137 12770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:02:22.980585 12770 catalog_manager.cc:1347] Generated new cluster ID: 6d3574c737304327a2d23dd43faf5e56
I20260926 03:02:22.980708 12770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:02:22.999809 12770 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:02:23.001670 12770 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:02:23.013995 12770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Generated new TSK 0
I20260926 03:02:23.014997 12770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:02:23.723071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:02:23 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:02:24.148229 12795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:24.148666 12795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:24.148871 12795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:24.160192 12795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:24.160324 12795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:24.160499 12795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:02:24.176329 12795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:50525/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:24.178431 12795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:24.180236 12795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:24.209069 12805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:24.209784 12806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:24.218254 12795 server_base.cc:1034] running on GCE node
W20260926 03:02:24.218636 12808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:24.219412 12795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:24.220664 12795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:24.221956 12795 hybrid_clock.cc:648] HybridClock initialized: now 1790391744221899 us; error 42 us; skew 500 ppm
I20260926 03:02:24.222378 12795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:24 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:24.226548 12795 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:02:24.228463 12795 webserver.cc:460] Webserver started at http://127.0.97.1:45457/ using document root <none> and password file <none>
I20260926 03:02:24.229297 12795 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:24.229425 12795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:24.229750 12795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:02:24 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:24.250025 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "6b7bc993eab946148a8522b50f498df2"
format_stamp: "Formatted at 2026-09-26 03:02:24 on dist-test-slave-fccw"
server_key: "6f4f71897c2a3299ac13438efca02e9d"
server_key_iv: "33e9fb739eba8810b53bdb4bae064591"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:24.253363 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6b7bc993eab946148a8522b50f498df2"
format_stamp: "Formatted at 2026-09-26 03:02:24 on dist-test-slave-fccw"
server_key: "6f4f71897c2a3299ac13438efca02e9d"
server_key_iv: "33e9fb739eba8810b53bdb4bae064591"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:24.260946 12795 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s
I20260926 03:02:24.293946 12815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:24.296104 12795 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.005s sys 0.011s
I20260926 03:02:24.296375 12795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "6b7bc993eab946148a8522b50f498df2"
format_stamp: "Formatted at 2026-09-26 03:02:24 on dist-test-slave-fccw"
server_key: "6f4f71897c2a3299ac13438efca02e9d"
server_key_iv: "33e9fb739eba8810b53bdb4bae064591"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:24.296725 12795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:24.328625 12795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:24.333370 12795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:24.333688 12795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:24.334743 12795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:24.336246 12795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:24.336359 12795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:24.336509 12795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:24.336598 12795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:24.413327 12795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39109
I20260926 03:02:24.413369 12927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39109 every 8 connection(s)
I20260926 03:02:24.415139 12795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:02:24.420940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 12795
I20260926 03:02:24.421238 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:02:24.422205 388 external_mini_cluster.cc:1428] Setting key 45655ba3560018b3863969a4d68a04b7
Sep 26 03:02:24 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 26 03:02:24 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:24.487190 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:24.435897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34983 (local address 127.0.97.62:46109)
0926 03:02:24.451996 (+ 16099us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:24.452003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:24.462364 (+ 10361us) server_negotiation.cc:407] Connection header received
0926 03:02:24.467725 (+ 5361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:24.467732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:24.467858 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:24.467967 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:24.471078 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:24.472347 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:24.474830 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:24.475451 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:24.475543 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:24.479882 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:24.479889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:24.479894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:24.479935 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:24.483481 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:24.484458 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:24.484465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:24.484469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:24.484625 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:24.485178 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:24.485184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:24.485187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:24.485523 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:24.485653 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:24.486564 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 03:02:24.486785 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15950,"thread_start_us":16102,"threads_started":1}
I20260926 03:02:24.493289 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:24.436750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.1:34983)
0926 03:02:24.462167 (+ 25417us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:24.462227 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:24.467401 (+ 5174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:24.469282 (+ 1881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:24.469297 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:24.469972 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:24.470846 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:24.470876 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:24.472556 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:24.472565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:24.474657 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:24.474671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:24.475867 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:24.475874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:24.476186 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:24.476927 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:24.476956 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:24.479622 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:24.483680 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:24.483689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:24.483707 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:24.484315 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:24.484774 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:24.484778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:24.484782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:24.485049 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:24.485736 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:24.485744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:24.486155 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 03:02:24.487600 (+ 1445us) client_negotiation.cc:240] Negotiation successful
0926 03:02:24.488186 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25024,"thread_start_us":26100,"threads_started":1}
I20260926 03:02:24.495169 12928 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260926 03:02:24.495499 12928 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:24.496232 12928 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:02:24.498430 12715 ts_manager.cc:194] Registered new tserver with Master: 6b7bc993eab946148a8522b50f498df2 (127.0.97.1:39109)
I20260926 03:02:24.500438 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260926 03:02:24.540122 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.114s user 0.000s sys 0.019s
I20260926 03:02:24.541070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:02:24 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:02:24.936954 12934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:24.937472 12934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:24.937641 12934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:24.949107 12934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:24.949302 12934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:24.949544 12934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:02:24.964636 12934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:50525/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:24.966378 12934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:24.968185 12934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:24.993839 12941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:24.994583 12942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:25.003827 12944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:25.004247 12934 server_base.cc:1034] running on GCE node
I20260926 03:02:25.005061 12934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:25.012449 12934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:25.013780 12934 hybrid_clock.cc:648] HybridClock initialized: now 1790391745013677 us; error 95 us; skew 500 ppm
I20260926 03:02:25.014212 12934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:25.018460 12934 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:02:25.029386 12934 webserver.cc:460] Webserver started at http://127.0.97.2:43117/ using document root <none> and password file <none>
I20260926 03:02:25.030318 12934 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:25.030447 12934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:25.030822 12934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:25.049399 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "43bd9dab85834f2aa27717a11615db95"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "5133e4c8642e3998d9bd971f95dea933"
server_key_iv: "fd749fe0c11aa03f37de62182e2556cb"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:25.050565 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "43bd9dab85834f2aa27717a11615db95"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "5133e4c8642e3998d9bd971f95dea933"
server_key_iv: "fd749fe0c11aa03f37de62182e2556cb"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:25.057899 12934 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 03:02:25.084107 12953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:25.088095 12934 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.004s sys 0.003s
I20260926 03:02:25.088275 12934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "43bd9dab85834f2aa27717a11615db95"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "5133e4c8642e3998d9bd971f95dea933"
server_key_iv: "fd749fe0c11aa03f37de62182e2556cb"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:25.088533 12934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:25.116279 12934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:25.121287 12934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:25.121593 12934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:25.122795 12934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:25.124461 12934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:25.124549 12934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:25.124655 12934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:25.124712 12934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:25.168738 12934 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33379
I20260926 03:02:25.168778 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33379 every 8 connection(s)
I20260926 03:02:25.170403 12934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:02:25.174706 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 12934
I20260926 03:02:25.174865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:02:25.175420 388 external_mini_cluster.cc:1428] Setting key 7b19cee24e0413b2f397bd35bff48319
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260926 03:02:25.216563 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:25.173250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41131 (local address 127.0.97.62:46109)
0926 03:02:25.189348 (+ 16098us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:25.189354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:25.189376 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:02:25.192097 (+ 2721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:25.192104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:25.192185 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:25.192282 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:25.199303 (+ 7021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.200428 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.203636 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.204309 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.204414 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:25.208849 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:25.208857 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:25.208862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:25.208902 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:25.212556 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.213490 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.213495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.213499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.213845 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.214393 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.214399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.214403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.214762 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:25.214891 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:25.215919 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0926 03:02:25.216169 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15975,"thread_start_us":16083,"threads_started":1}
I20260926 03:02:25.218948 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:25.174014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.2:41131)
0926 03:02:25.188778 (+ 14764us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:25.188837 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:25.191631 (+ 2794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:25.197423 (+ 5792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:25.197436 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:25.198187 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:25.199075 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:25.199103 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.200804 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.200809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:25.203440 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:25.203456 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.204744 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.204752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:25.205051 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:25.205885 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:25.205915 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:25.208590 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:25.212722 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:25.212731 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:25.212752 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:25.213354 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:25.213997 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:25.214002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:25.214006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:25.214264 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:25.214990 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:25.214998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:25.215491 (+ 493us) client_negotiation.cc:769] Sending connection context
0926 03:02:25.216880 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0926 03:02:25.217457 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14436,"thread_start_us":14206,"threads_started":1}
I20260926 03:02:25.220641 13071 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260926 03:02:25.220966 13071 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:25.221843 13071 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260926 03:02:25.223369 12715 ts_manager.cc:194] Registered new tserver with Master: 43bd9dab85834f2aa27717a11615db95 (127.0.97.2:33379)
I20260926 03:02:25.224205 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:02:25.244164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:02:25.504091 12928 heartbeater.cc:503] Master 127.0.97.62:46109 was elected leader, sending a full tablet report...
W20260926 03:02:25.619585 13077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:25.619897 13077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:25.620069 13077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:25.626912 13077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:25.627051 13077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:25.627236 13077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:02:25.636071 13077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:50525/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:25.637356 13077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:25.638597 13077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:25.650974 13082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:25.650974 13083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:25.650828 13085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:25.652076 13077 server_base.cc:1034] running on GCE node
I20260926 03:02:25.652652 13077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:25.653577 13077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:25.654794 13077 hybrid_clock.cc:648] HybridClock initialized: now 1790391745654738 us; error 47 us; skew 500 ppm
I20260926 03:02:25.655097 13077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:25.658028 13077 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:02:25.659376 13077 webserver.cc:460] Webserver started at http://127.0.97.3:41539/ using document root <none> and password file <none>
I20260926 03:02:25.660005 13077 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:25.660146 13077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:25.660440 13077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:25.677619 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "524533dc43e11ac6c2e14003456d00fe"
server_key_iv: "a4dc6b623327cb5bc7e657de2d96617e"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:25.678686 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "524533dc43e11ac6c2e14003456d00fe"
server_key_iv: "a4dc6b623327cb5bc7e657de2d96617e"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:25.685318 13077 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 03:02:25.699628 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:25.701200 13077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:02:25.701372 13077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "524533dc43e11ac6c2e14003456d00fe"
server_key_iv: "a4dc6b623327cb5bc7e657de2d96617e"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:25.701586 13077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:25.715459 13077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:25.719161 13077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:25.719455 13077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:25.720381 13077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:25.721760 13077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:25.721828 13077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:25.721908 13077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:25.721954 13077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:25.753517 13077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42013
I20260926 03:02:25.753551 13204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42013 every 8 connection(s)
I20260926 03:02:25.754771 13077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:02:25.764174 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13077
I20260926 03:02:25.764365 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:02:25.764840 388 external_mini_cluster.cc:1428] Setting key 786f19f669cb30ece8cb6a296f472ad4
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:25.773823 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:25.757215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39943 (local address 127.0.97.62:46109)
0926 03:02:25.757688 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:25.757693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:25.758532 (+ 839us) server_negotiation.cc:407] Connection header received
0926 03:02:25.760291 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:25.760298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:25.760402 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:25.760530 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:25.761988 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.762705 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.764216 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.764761 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.764900 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:25.768640 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:25.768645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:25.768648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:25.768685 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:25.771242 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.771905 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.771909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.771911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.772025 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.772391 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.772394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.772395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.772632 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:25.772735 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:25.773403 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 03:02:25.773563 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":269,"threads_started":1}
I20260926 03:02:25.774806 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:25.757711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.3:39943)
0926 03:02:25.758371 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:25.758417 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:25.759986 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:25.760749 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:25.760757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:25.761269 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:25.761828 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:25.761843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.762855 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.762860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:25.764044 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:25.764077 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.764988 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.765021 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:25.765345 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:25.765983 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:25.766003 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:25.768451 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:25.771367 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:25.771373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:25.771385 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:25.771786 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:25.772120 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:25.772123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:25.772125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:25.772296 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:25.772732 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:25.772737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:25.773056 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 03:02:25.773312 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 03:02:25.773654 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":293,"threads_started":1}
I20260926 03:02:25.776145 13205 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260926 03:02:25.776525 13205 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:25.777269 13205 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260926 03:02:25.778656 12715 ts_manager.cc:194] Registered new tserver with Master: 9b76a5c3645c4ff399e42d5bf9b14a1b (127.0.97.3:42013)
I20260926 03:02:25.779737 12715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39943
I20260926 03:02:25.780471 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:02:25.782166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 12795
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:02:25.798076 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:25.782952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.97.1:39109)
0926 03:02:25.783507 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:25.783513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:25.783536 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:02:25.783586 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:25.783592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:25.783768 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:25.783887 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:25.785368 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.786344 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.787786 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.788310 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.788403 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:25.792495 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:25.792506 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:25.792518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:25.792553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:25.795493 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.796092 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.796102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.796104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.796237 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.796615 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.796618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.796620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.796911 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:25.797214 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:25.797479 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 03:02:25.797665 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":269,"threads_started":1}
I20260926 03:02:25.799356 12923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:25.882436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 12795
I20260926 03:02:25.898976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 12934
Sep 26 03:02:25 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:02:25.915728 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:25.899881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.0.97.2:33379)
0926 03:02:25.900402 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:25.900412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:25.900441 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:02:25.900509 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:25.900518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:25.900780 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:25.900946 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:25.902172 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.903663 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.904941 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:25.905803 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:25.905916 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:25.910158 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:25.910167 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:25.910183 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:25.910232 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:25.912852 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.913778 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.913785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.913788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.913932 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:25.914259 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:25.914262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:25.914264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:25.914594 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:25.914797 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:25.915140 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 03:02:25.915340 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":264,"threads_started":1}
I20260926 03:02:25.916936 13066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:25.997781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 12934
I20260926 03:02:26.013141 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13077
Sep 26 03:02:26 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:02:26.029640 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:26.014209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.97.3:42013)
0926 03:02:26.014856 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:26.014863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:26.014891 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:02:26.014945 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:26.014951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:26.015242 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:26.015448 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:26.016852 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:26.017983 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:26.019334 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:26.019949 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:26.020085 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:26.024478 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:26.024488 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:26.024502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:26.024536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:26.027308 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:26.027903 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:26.027909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:26.027912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:26.028026 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:26.028330 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:26.028333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:26.028335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:26.028634 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:26.028849 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:26.029156 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 03:02:26.029348 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":362,"threads_started":1}
I20260926 03:02:26.030941 13200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:26.116600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13077
I20260926 03:02:26.132067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 12683
I20260926 03:02:26.133368 12745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:26.238226 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 12683
I20260926 03:02:26.259271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--webserver_interface=127.0.97.62
--webserver_port=42853
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46109
--ranger_config_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:26.576122 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:26.576475 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:26.576651 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:26.583894 13217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:26.584070 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:26.584199 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:26.584307 13217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:26.584409 13217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:26.594584 13217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46109
--ranger_config_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:26.595888 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:26.597218 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:26.608961 13223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:26.609078 13222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:26.609032 13225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:26.609917 13217 server_base.cc:1034] running on GCE node
I20260926 03:02:26.610570 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:26.611868 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:26.613124 13217 hybrid_clock.cc:648] HybridClock initialized: now 1790391746613066 us; error 62 us; skew 500 ppm
I20260926 03:02:26.613454 13217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:26 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:26.616465 13217 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:02:26.618013 13217 webserver.cc:460] Webserver started at http://127.0.97.62:42853/ using document root <none> and password file <none>
I20260926 03:02:26.618693 13217 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:26.618808 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:02:26 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:26.676468 13217 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:02:26.680105 13233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:26.681780 13217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:02:26.681968 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "ff367b148cce49d9b7f9bf434523fcb6"
format_stamp: "Formatted at 2026-09-26 03:02:22 on dist-test-slave-fccw"
server_key: "cf72ddb4ef36bf275593a4599653efa1"
server_key_iv: "c8d7eeec0fff791949390670287b77e5"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:26.683267 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:26.705874 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:26.710528 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:26.710819 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:26.730329 13217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46109
I20260926 03:02:26.730347 13284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46109 every 8 connection(s)
I20260926 03:02:26.731730 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:02:26.738468 13285 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:02:26.740577 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13217
I20260926 03:02:26.741596 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39109
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46109
--webserver_port=45457
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:02:26.742504 13285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Bootstrap starting.
I20260926 03:02:26.762562 13285 log.cc:824] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:26.771080 13285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:02:26.771600 13285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Bootstrap complete.
I20260926 03:02:26.780488 13285 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:26.781399 13285 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff367b148cce49d9b7f9bf434523fcb6, State: Initialized, Role: FOLLOWER
I20260926 03:02:26.782027 13285 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:26.782259 13285 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:02:26.782377 13285 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:02:26.782574 13285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:02:26.786329 13285 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:26.786801 13285 leader_election.cc:302] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff367b148cce49d9b7f9bf434523fcb6; no voters:
I20260926 03:02:26.787436 13285 leader_election.cc:288] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:02:26.787563 13289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:02:26.788643 13289 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [term 2 LEADER]: Becoming Leader. State: Replica: ff367b148cce49d9b7f9bf434523fcb6, State: Running, Role: LEADER
I20260926 03:02:26.789294 13289 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } }
I20260926 03:02:26.789853 13285 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:26.791977 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff367b148cce49d9b7f9bf434523fcb6. Latest consensus state: current_term: 2 leader_uuid: "ff367b148cce49d9b7f9bf434523fcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260926 03:02:26.792397 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:26.793066 13290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ff367b148cce49d9b7f9bf434523fcb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff367b148cce49d9b7f9bf434523fcb6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46109 } } }
I20260926 03:02:26.793401 13290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:26.797716 13295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:02:26.804245 13295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:02:26.806088 13295 catalog_manager.cc:1259] Loaded cluster ID: 6d3574c737304327a2d23dd43faf5e56
I20260926 03:02:26.806267 13295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:02:26.811448 13295 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:02:26.813576 13295 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Loaded TSK: 0
I20260926 03:02:26.815435 13295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:02:26.819262 13285 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 03:02:27.200649 13287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:27.201102 13287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:27.201328 13287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:27.213165 13287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:27.213363 13287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:27.213599 13287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:02:27.229365 13287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45457
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:27.231297 13287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:27.233079 13287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:27.280086 13322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:27.280110 13323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:27.281786 13325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:27.287091 13287 server_base.cc:1034] running on GCE node
I20260926 03:02:27.288321 13287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:27.289839 13287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:27.291203 13287 hybrid_clock.cc:648] HybridClock initialized: now 1790391747291122 us; error 60 us; skew 500 ppm
I20260926 03:02:27.291683 13287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:27 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:27.296501 13287 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:02:27.302440 13287 webserver.cc:460] Webserver started at http://127.0.97.1:45457/ using document root <none> and password file <none>
I20260926 03:02:27.303211 13287 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:27.303388 13287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:02:27 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:27.333505 13287 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 03:02:27.337571 13333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:27.339412 13287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:02:27.339653 13287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "6b7bc993eab946148a8522b50f498df2"
format_stamp: "Formatted at 2026-09-26 03:02:24 on dist-test-slave-fccw"
server_key: "6f4f71897c2a3299ac13438efca02e9d"
server_key_iv: "33e9fb739eba8810b53bdb4bae064591"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:27.341233 13287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:27.412572 13287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:27.418298 13287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:27.418643 13287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:27.433499 13287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:27.435400 13287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:27.435549 13287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:27.435725 13287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:27.435842 13287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:27.496259 13287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39109
I20260926 03:02:27.496308 13446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39109 every 8 connection(s)
I20260926 03:02:27.498327 13287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:02:27.504958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13287
I20260926 03:02:27.505817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33379
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46109
--webserver_port=43117
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:02:27 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:27.530751 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:27.501938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.1:57529)
0926 03:02:27.502838 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:27.502894 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:27.504602 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:27.507078 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:27.507091 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:27.507699 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:27.508426 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:27.508451 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:27.510823 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:27.510832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:27.513245 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:27.513275 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:27.514749 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:27.514759 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:27.515079 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:27.515858 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:27.515894 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:27.519066 (+ 3172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:27.524463 (+ 5397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:27.524476 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:27.524497 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:27.525260 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:27.525883 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:27.525891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:27.525899 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:27.526207 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:27.527403 (+ 1196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:27.527416 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:27.527912 (+ 496us) client_negotiation.cc:769] Sending connection context
0926 03:02:27.528272 (+ 360us) client_negotiation.cc:240] Negotiation successful
0926 03:02:27.529002 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":416,"threads_started":1}
I20260926 03:02:27.532626 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:27.501885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57529 (local address 127.0.97.62:46109)
0926 03:02:27.502789 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:27.502808 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:27.503024 (+ 216us) server_negotiation.cc:407] Connection header received
0926 03:02:27.505437 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:27.505459 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:27.505954 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:27.506373 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:27.508656 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:27.510621 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:27.513430 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:27.514484 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:27.514673 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:27.519422 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:27.519436 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:27.519459 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:27.519518 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:27.524271 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:27.525468 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:27.525481 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:27.525494 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:27.525732 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:27.526391 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:27.526399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:27.526406 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:27.527206 (+ 800us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:27.530724 (+ 3518us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:27.531028 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:02:27.531437 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":377,"threads_started":1}
I20260926 03:02:27.536474 13447 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260926 03:02:27.536818 13447 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:27.537667 13447 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
W20260926 03:02:27.540606 13447 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46109 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:02:27.540815 13447 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:27.541069 13447 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260926 03:02:27.542315 13447 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:27.542589 13447 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
W20260926 03:02:27.543751 13447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:02:27.895430 13451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:27.895788 13451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:27.895982 13451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:27.903707 13451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:27.903802 13451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:27.903990 13451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:02:27.913776 13451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43117
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:27.915033 13451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:27.916384 13451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:27.931389 13463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:27.931452 13464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:27.931903 13466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:27.932521 13451 server_base.cc:1034] running on GCE node
I20260926 03:02:27.933254 13451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:27.934268 13451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:27.935497 13451 hybrid_clock.cc:648] HybridClock initialized: now 1790391747935441 us; error 51 us; skew 500 ppm
I20260926 03:02:27.935796 13451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:27 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:27.944648 13451 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:02:27.946273 13451 webserver.cc:460] Webserver started at http://127.0.97.2:43117/ using document root <none> and password file <none>
I20260926 03:02:27.947089 13451 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:27.947258 13451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:02:27 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:27.980926 13451 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.008s
I20260926 03:02:27.985687 13476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:27.987511 13451 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:02:27.987686 13451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "43bd9dab85834f2aa27717a11615db95"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "5133e4c8642e3998d9bd971f95dea933"
server_key_iv: "fd749fe0c11aa03f37de62182e2556cb"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:27.989176 13451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:28.018802 13451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:28.023859 13451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:28.024139 13451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:28.025349 13451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:28.026996 13451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:28.027108 13451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:28.027259 13451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:28.027343 13451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:28.118450 13451 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33379
I20260926 03:02:28.118541 13590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33379 every 8 connection(s)
I20260926 03:02:28.120303 13451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:02:28.127940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13451
I20260926 03:02:28.128835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:50525/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42013
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46109
--webserver_port=41539
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:02:28 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:28.157763 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.123648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.2:32951)
0926 03:02:28.132855 (+ 9207us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:28.132909 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:28.136358 (+ 3449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:28.137851 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:28.137863 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:28.139049 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:28.139798 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.139822 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.141443 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.141448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.143569 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.143598 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.144896 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.144905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.145243 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.146070 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:28.146106 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:28.148807 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:28.152889 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:28.152898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:28.152915 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:28.153518 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:28.153990 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:28.153994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:28.153998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:28.154240 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:28.154976 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:28.154984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:28.155479 (+ 495us) client_negotiation.cc:769] Sending connection context
0926 03:02:28.155760 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 03:02:28.156304 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8888,"spinlock_wait_cycles":126720,"thread_start_us":9280,"threads_started":1}
I20260926 03:02:28.159766 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.123057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32951 (local address 127.0.97.62:46109)
0926 03:02:28.137050 (+ 13993us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:28.137058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:28.137082 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:02:28.137149 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:28.137156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:28.137246 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:28.137400 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:28.140023 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.141241 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.143784 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.144608 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.145716 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.149098 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:28.149105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:28.149110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:28.149147 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:28.152723 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.153688 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.153693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.153696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.153856 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.154407 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.154412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.154415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.154825 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:28.158973 (+ 4148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:28.159111 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 03:02:28.159383 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13826,"thread_start_us":13712,"threads_started":1}
I20260926 03:02:28.161397 13591 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260926 03:02:28.161736 13591 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:28.162611 13591 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260926 03:02:28.164740 13250 ts_manager.cc:194] Registered new tserver with Master: 43bd9dab85834f2aa27717a11615db95 (127.0.97.2:33379)
I20260926 03:02:28.167042 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32951
Sep 26 03:02:28 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790391748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:28.543932 13447 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:28.544401 13447 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260926 03:02:28.546088 13250 ts_manager.cc:194] Registered new tserver with Master: 6b7bc993eab946148a8522b50f498df2 (127.0.97.1:39109)
I20260926 03:02:28.547235 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57529
W20260926 03:02:28.571233 13595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:28.571705 13595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:28.571915 13595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 26 03:02:28 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260926 03:02:28.582917 13595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:28.583082 13595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:28.583261 13595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:02:28.596777 13595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:50525/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41539
--tserver_master_addrs=127.0.97.62:46109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:28.598531 13595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:28.600291 13595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:28.639652 13601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:28.645807 13602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:28.666494 13595 server_base.cc:1034] running on GCE node
W20260926 03:02:28.667660 13604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:28.668692 13595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:28.672734 13595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:28.674038 13595 hybrid_clock.cc:648] HybridClock initialized: now 1790391748673944 us; error 96 us; skew 500 ppm
I20260926 03:02:28.674449 13595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:28 dist-test-slave-fccw krb5kdc[12239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:28.678700 13595 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:02:28.685251 13595 webserver.cc:460] Webserver started at http://127.0.97.3:41539/ using document root <none> and password file <none>
I20260926 03:02:28.686236 13595 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:28.686443 13595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 03:02:28 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:28.708297 13595 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:02:28.711818 13612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:28.713725 13595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 03:02:28.713896 13595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b"
format_stamp: "Formatted at 2026-09-26 03:02:25 on dist-test-slave-fccw"
server_key: "524533dc43e11ac6c2e14003456d00fe"
server_key_iv: "a4dc6b623327cb5bc7e657de2d96617e"
server_key_version: "kuduclusterkey@0"
I20260926 03:02:28.715207 13595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:28.735129 13595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:28.740047 13595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:28.740422 13595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:28.741636 13595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:28.743373 13595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:28.743495 13595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:28.743629 13595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:28.743721 13595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:28.792905 13595 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42013
I20260926 03:02:28.792953 13724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42013 every 8 connection(s)
I20260926 03:02:28.794869 13595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:02:28.796551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13595
Sep 26 03:02:28 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:28.828963 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.797564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.0.97.62:46109)
0926 03:02:28.801897 (+ 4333us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:28.801903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:28.801922 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:02:28.802005 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:28.802010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:28.802105 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:28.802218 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:28.805346 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.806466 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.813960 (+ 7494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.814712 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.815630 (+ 918us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.819680 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:28.819686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:28.819691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:28.819734 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:28.823210 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.824536 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.824541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.824546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.824699 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.826415 (+ 1716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.826420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.826424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.826752 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:28.827035 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:28.828360 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0926 03:02:28.828591 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":4226,"thread_start_us":4122,"threads_started":1}
I20260926 03:02:28.829671 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.800465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59137 (local address 127.0.97.62:46109)
0926 03:02:28.811639 (+ 11174us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:28.811645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:28.811662 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:02:28.811708 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:28.811713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:28.811784 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:28.811891 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:28.813966 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.815337 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.817571 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.818206 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.818307 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.822607 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:28.822613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:28.822617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:28.822661 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:28.826252 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.827098 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.827103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.827106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.827229 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.827759 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.827764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.827768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.828077 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:28.828219 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:28.829154 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 03:02:28.829317 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":11049,"thread_start_us":10901,"threads_started":1}
I20260926 03:02:28.832547 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.799785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46109 (local address 127.0.97.3:59137)
0926 03:02:28.808520 (+ 8735us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:28.808567 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:28.810945 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:28.812190 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:28.812202 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:28.812745 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:28.813701 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.813728 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.815539 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.815545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.817411 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.817424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.818575 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.818580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.818908 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.819601 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:28.819631 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:28.822335 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:28.826399 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:28.826407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:28.826423 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:28.826978 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:28.827358 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:28.827363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:28.827366 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:28.827625 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:28.828196 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:28.828203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:28.828563 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 03:02:28.828840 (+ 277us) client_negotiation.cc:240] Negotiation successful
0926 03:02:28.830795 (+ 1955us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8420,"thread_start_us":7417,"threads_started":1}
I20260926 03:02:28.837294 13725 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46109
I20260926 03:02:28.837613 13725 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:28.838433 13725 heartbeater.cc:511] Master 127.0.97.62:46109 requested a full tablet report, sending...
I20260926 03:02:28.840054 13250 ts_manager.cc:194] Registered new tserver with Master: 9b76a5c3645c4ff399e42d5bf9b14a1b (127.0.97.3:42013)
I20260926 03:02:28.841223 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59137
I20260926 03:02:28.846395 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:02:28.870323 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.850151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.0.97.62:46109)
0926 03:02:28.850322 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:28.850328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:28.853545 (+ 3217us) server_negotiation.cc:407] Connection header received
0926 03:02:28.853714 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:28.853718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:28.853809 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:28.853915 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:28.855803 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.856874 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.858894 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.859528 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.859633 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.863270 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:28.863278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:28.863283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:28.863333 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:28.866792 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.867572 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.867577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.867580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.867718 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:28.868177 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:28.868182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:28.868185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:28.868591 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:28.868850 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:28.869812 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 03:02:28.870023 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:02:28.876195 13250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:02:28.887722 13250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 03:02:28 dist-test-slave-fccw krb5kdc[12239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260926 03:02:28.931684 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.907569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42013 (local address 127.0.0.1:50714)
0926 03:02:28.920699 (+ 13130us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:28.920742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:28.920949 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:28.923635 (+ 2686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:28.923643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:28.923673 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:28.924084 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.924101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.926451 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.926457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.928663 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.928677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.930714 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.930720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.930938 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.930996 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:02:28.931142 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 03:02:28.931331 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12282,"thread_start_us":12145,"threads_started":1}
I20260926 03:02:28.932729 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.907751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.0.97.3:42013)
0926 03:02:28.922679 (+ 14928us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:28.922688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:28.922723 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:02:28.922818 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:28.922826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:28.923114 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:28.923301 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:28.924596 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.926290 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.928840 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.930222 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.930337 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.930559 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:28.932236 (+ 1677us) server_negotiation.cc:299] Negotiation successful
0926 03:02:28.932408 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14756,"thread_start_us":16500,"threads_started":1}
I20260926 03:02:28.942445 13660 tablet_service.cc:1467] Processing CreateTablet for tablet 64c505f37a4640f18dd6e4f8d8a09b90 (DEFAULT_TABLE table=test-table [id=564a0514e9dc476b892b80dd0eaec7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:28.944058 13660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c505f37a4640f18dd6e4f8d8a09b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:28.953893 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.923918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37752)
0926 03:02:28.938270 (+ 14352us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:28.938292 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:28.938433 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:28.940299 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:28.940304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:28.940329 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:28.940694 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.940705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.943245 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.943253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.947302 (+ 4049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.947317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.953091 (+ 5774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.953097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.953285 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.953320 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:02:28.953420 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:02:28.953556 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14228,"thread_start_us":14071,"threads_started":1}
I20260926 03:02:28.954867 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.909744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33379 (local address 127.0.0.1:52042)
0926 03:02:28.931929 (+ 22185us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:28.931950 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:28.932050 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:28.941174 (+ 9124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:28.941179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:28.941203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:28.941367 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.941379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.949447 (+ 8068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.949453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.951245 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:28.951261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.954181 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.954186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:28.954372 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.954423 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 03:02:28.954485 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 03:02:28.954613 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22080,"thread_start_us":44419,"threads_started":1}
I20260926 03:02:28.955840 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.910282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.0.97.2:33379)
0926 03:02:28.938890 (+ 28608us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:28.938907 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:28.938941 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:02:28.939034 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:28.939043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:28.939330 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:28.945119 (+ 5789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:28.945399 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.946855 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.951463 (+ 4608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.952841 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.955095 (+ 2254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.955278 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:28.955347 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 03:02:28.955534 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28453,"thread_start_us":36888,"threads_started":1}
I20260926 03:02:28.956615 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:28.921111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.0.97.1:39109)
0926 03:02:28.937848 (+ 16737us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:28.937859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:28.939549 (+ 1690us) server_negotiation.cc:407] Connection header received
0926 03:02:28.939646 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:28.939656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:28.939875 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:28.940054 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:28.941543 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.943061 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.947509 (+ 4448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:28.948972 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:28.949109 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:28.949330 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:28.956101 (+ 6771us) server_negotiation.cc:299] Negotiation successful
0926 03:02:28.956269 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16537,"thread_start_us":16381,"threads_started":1}
I20260926 03:02:28.959909 13524 tablet_service.cc:1467] Processing CreateTablet for tablet 64c505f37a4640f18dd6e4f8d8a09b90 (DEFAULT_TABLE table=test-table [id=564a0514e9dc476b892b80dd0eaec7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:28.963595 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 64c505f37a4640f18dd6e4f8d8a09b90 (DEFAULT_TABLE table=test-table [id=564a0514e9dc476b892b80dd0eaec7fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:28.964932 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c505f37a4640f18dd6e4f8d8a09b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:28.965546 13524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64c505f37a4640f18dd6e4f8d8a09b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:28.988749 13748 tablet_bootstrap.cc:492] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Bootstrap starting.
I20260926 03:02:28.996214 13748 tablet_bootstrap.cc:654] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:28.997991 13748 log.cc:824] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:29.021377 13751 tablet_bootstrap.cc:492] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Bootstrap starting.
I20260926 03:02:29.029412 13751 tablet_bootstrap.cc:654] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:29.031033 13751 log.cc:824] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:29.034850 13748 tablet_bootstrap.cc:492] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: No bootstrap required, opened a new log
I20260926 03:02:29.035187 13748 ts_tablet_manager.cc:1397] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.013s
I20260926 03:02:29.035684 13751 tablet_bootstrap.cc:492] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: No bootstrap required, opened a new log
I20260926 03:02:29.036021 13751 ts_tablet_manager.cc:1397] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20260926 03:02:29.036474 13749 tablet_bootstrap.cc:492] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Bootstrap starting.
I20260926 03:02:29.042254 13749 tablet_bootstrap.cc:654] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:29.043654 13749 log.cc:824] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:29.048203 13751 raft_consensus.cc:348] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.048658 13751 raft_consensus.cc:374] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:29.048763 13751 raft_consensus.cc:729] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43bd9dab85834f2aa27717a11615db95, State: Initialized, Role: FOLLOWER
I20260926 03:02:29.049515 13751 consensus_queue.cc:260] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.054200 13749 tablet_bootstrap.cc:492] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: No bootstrap required, opened a new log
I20260926 03:02:29.054534 13749 ts_tablet_manager.cc:1397] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.007s
I20260926 03:02:29.054610 13751 ts_tablet_manager.cc:1428] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Time spent starting tablet: real 0.018s user 0.012s sys 0.000s
I20260926 03:02:29.056792 13748 raft_consensus.cc:348] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.057289 13748 raft_consensus.cc:374] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:29.057468 13748 raft_consensus.cc:729] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b76a5c3645c4ff399e42d5bf9b14a1b, State: Initialized, Role: FOLLOWER
I20260926 03:02:29.058207 13748 consensus_queue.cc:260] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.059593 13591 heartbeater.cc:503] Master 127.0.97.62:46109 was elected leader, sending a full tablet report...
I20260926 03:02:29.066424 13749 raft_consensus.cc:348] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.066983 13749 raft_consensus.cc:374] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:29.067117 13749 raft_consensus.cc:729] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7bc993eab946148a8522b50f498df2, State: Initialized, Role: FOLLOWER
I20260926 03:02:29.067940 13749 consensus_queue.cc:260] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.077515 13447 heartbeater.cc:503] Master 127.0.97.62:46109 was elected leader, sending a full tablet report...
I20260926 03:02:29.077764 13749 ts_tablet_manager.cc:1428] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Time spent starting tablet: real 0.023s user 0.018s sys 0.002s
I20260926 03:02:29.089852 13725 heartbeater.cc:503] Master 127.0.97.62:46109 was elected leader, sending a full tablet report...
I20260926 03:02:29.090770 13748 ts_tablet_manager.cc:1428] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Time spent starting tablet: real 0.055s user 0.028s sys 0.007s
W20260926 03:02:29.139788 13592 tablet.cc:2369] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:02:29.251081 13448 tablet.cc:2369] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:02:29.307921 13726 tablet.cc:2369] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:02:29.310940 13755 raft_consensus.cc:482] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:29.311234 13755 raft_consensus.cc:504] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.312769 13755 leader_election.cc:288] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7bc993eab946148a8522b50f498df2 (127.0.97.1:39109), 43bd9dab85834f2aa27717a11615db95 (127.0.97.2:33379)
I20260926 03:02:29.320173 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.313464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55791 (local address 127.0.97.1:39109)
0926 03:02:29.313719 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:29.313728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:29.313756 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:02:29.313836 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:29.313842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:29.313915 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:29.314021 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:29.314424 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.315525 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.318003 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.319223 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.319351 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.319453 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:29.319754 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 03:02:29.319879 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 03:02:29.320200 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.313145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.97.3:55791)
0926 03:02:29.313408 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.313430 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:29.313558 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:29.314077 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:29.314081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:29.314100 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:29.314292 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:29.314300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.315676 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.315680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:29.317837 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:29.317851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.319391 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.319396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:29.319570 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.319620 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 03:02:29.319715 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:02:29.319923 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":27}
I20260926 03:02:29.320480 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.313454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39631 (local address 127.0.97.2:33379)
0926 03:02:29.313642 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:29.313647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:29.314104 (+ 457us) server_negotiation.cc:407] Connection header received
0926 03:02:29.314242 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:29.314246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:29.314311 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:29.314423 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:29.314875 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.315881 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.318189 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.319406 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.319567 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.319667 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:29.320081 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 03:02:29.320222 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:02:29.320598 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.313197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33379 (local address 127.0.97.3:39631)
0926 03:02:29.313951 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.313964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:29.314098 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:29.314454 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:29.314457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:29.314476 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:29.314712 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:29.314719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.315998 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.316003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:29.318044 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:29.318053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.319667 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.319674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:29.319870 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.319902 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:02:29.320043 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:02:29.320164 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":19,"thread_start_us":562,"threads_started":1}
I20260926 03:02:29.321465 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c505f37a4640f18dd6e4f8d8a09b90" candidate_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7bc993eab946148a8522b50f498df2" is_pre_election: true
I20260926 03:02:29.321640 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c505f37a4640f18dd6e4f8d8a09b90" candidate_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bd9dab85834f2aa27717a11615db95" is_pre_election: true
I20260926 03:02:29.322088 13402 raft_consensus.cc:2454] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b76a5c3645c4ff399e42d5bf9b14a1b in term 0.
I20260926 03:02:29.322098 13546 raft_consensus.cc:2454] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b76a5c3645c4ff399e42d5bf9b14a1b in term 0.
I20260926 03:02:29.322791 13616 leader_election.cc:302] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43bd9dab85834f2aa27717a11615db95, 9b76a5c3645c4ff399e42d5bf9b14a1b; no voters:
I20260926 03:02:29.323190 13755 raft_consensus.cc:2781] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:02:29.323347 13755 raft_consensus.cc:482] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:02:29.323474 13755 raft_consensus.cc:3037] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:29.325726 13755 raft_consensus.cc:504] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.326284 13755 leader_election.cc:288] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [CANDIDATE]: Term 1 election: Requested vote from peers 6b7bc993eab946148a8522b50f498df2 (127.0.97.1:39109), 43bd9dab85834f2aa27717a11615db95 (127.0.97.2:33379)
I20260926 03:02:29.326705 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c505f37a4640f18dd6e4f8d8a09b90" candidate_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7bc993eab946148a8522b50f498df2"
I20260926 03:02:29.326761 13546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64c505f37a4640f18dd6e4f8d8a09b90" candidate_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43bd9dab85834f2aa27717a11615db95"
I20260926 03:02:29.326893 13402 raft_consensus.cc:3037] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:29.327036 13546 raft_consensus.cc:3037] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:29.329172 13402 raft_consensus.cc:2454] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b76a5c3645c4ff399e42d5bf9b14a1b in term 1.
I20260926 03:02:29.329553 13614 leader_election.cc:302] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7bc993eab946148a8522b50f498df2, 9b76a5c3645c4ff399e42d5bf9b14a1b; no voters:
I20260926 03:02:29.329696 13546 raft_consensus.cc:2454] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b76a5c3645c4ff399e42d5bf9b14a1b in term 1.
I20260926 03:02:29.329815 13755 raft_consensus.cc:2781] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:02:29.330328 13755 raft_consensus.cc:686] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 1 LEADER]: Becoming Leader. State: Replica: 9b76a5c3645c4ff399e42d5bf9b14a1b, State: Running, Role: LEADER
I20260926 03:02:29.331045 13755 consensus_queue.cc:237] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } }
I20260926 03:02:29.335251 13250 catalog_manager.cc:5495] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b76a5c3645c4ff399e42d5bf9b14a1b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b76a5c3645c4ff399e42d5bf9b14a1b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 } health_report { overall_health: UNKNOWN } } }
I20260926 03:02:29.419546 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.414372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.0.97.3:42013)
0926 03:02:29.414553 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:29.414557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:29.414714 (+ 157us) server_negotiation.cc:407] Connection header received
0926 03:02:29.414879 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:29.414882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:29.414960 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:29.415092 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:02:29.415413 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.416270 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.417639 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.418021 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.418114 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.418538 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:02:29.419053 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:02:29.419164 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:29.419227 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 03:02:29.419286 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:02:29.427366 13402 raft_consensus.cc:1261] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 1 FOLLOWER]: Refusing update from remote peer 9b76a5c3645c4ff399e42d5bf9b14a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:29.427369 13546 raft_consensus.cc:1261] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 1 FOLLOWER]: Refusing update from remote peer 9b76a5c3645c4ff399e42d5bf9b14a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:29.428333 13765 consensus_queue.cc:1035] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "43bd9dab85834f2aa27717a11615db95" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:29.428676 13755 consensus_queue.cc:1035] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7bc993eab946148a8522b50f498df2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:29.440697 13769 mvcc.cc:205] Tried to move back new op lower bound from 7333444605645168640 to 7333444605263720448. Current Snapshot: MvccSnapshot[applied={T|T < 7333444605645168640}]
I20260926 03:02:29.458395 13249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:02:29.458693 13249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:02:29.462172 13249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff367b148cce49d9b7f9bf434523fcb6: Sending DeleteTablet for 3 replicas of tablet 64c505f37a4640f18dd6e4f8d8a09b90
I20260926 03:02:29.463140 13660 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c505f37a4640f18dd6e4f8d8a09b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50714
I20260926 03:02:29.463296 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c505f37a4640f18dd6e4f8d8a09b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37752
I20260926 03:02:29.463784 13524 tablet_service.cc:1514] Processing DeleteTablet for tablet 64c505f37a4640f18dd6e4f8d8a09b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52042
I20260926 03:02:29.466104 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13287
I20260926 03:02:29.477077 13781 tablet_replica.cc:330] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: stopping tablet replica
I20260926 03:02:29.477295 13779 tablet_replica.cc:330] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: stopping tablet replica
I20260926 03:02:29.477864 13779 raft_consensus.cc:2229] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:02:29.478727 13779 raft_consensus.cc:2258] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:29.479602 13780 tablet_replica.cc:330] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: stopping tablet replica
I20260926 03:02:29.479960 13780 raft_consensus.cc:2229] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:02:29.480515 13780 raft_consensus.cc:2258] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:29.477567 13781 raft_consensus.cc:2229] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:02:29.482093 13779 ts_tablet_manager.cc:1905] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:29.482651 13780 ts_tablet_manager.cc:1905] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:29.483103 13781 raft_consensus.cc:2258] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:29.487555 13781 ts_tablet_manager.cc:1905] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:29.489945 13779 ts_tablet_manager.cc:1918] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:29.490145 13779 log.cc:1196] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/64c505f37a4640f18dd6e4f8d8a09b90
I20260926 03:02:29.490746 13779 ts_tablet_manager.cc:1939] T 64c505f37a4640f18dd6e4f8d8a09b90 P 9b76a5c3645c4ff399e42d5bf9b14a1b: Deleting consensus metadata
I20260926 03:02:29.492574 13235 catalog_manager.cc:4841] TS 9b76a5c3645c4ff399e42d5bf9b14a1b (127.0.97.3:42013): tablet 64c505f37a4640f18dd6e4f8d8a09b90 (table test-table [id=564a0514e9dc476b892b80dd0eaec7fb]) successfully deleted
I20260926 03:02:29.493992 13780 ts_tablet_manager.cc:1918] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:29.497392 13781 ts_tablet_manager.cc:1918] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:29.497581 13781 log.cc:1196] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/64c505f37a4640f18dd6e4f8d8a09b90
I20260926 03:02:29.494933 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.467095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.0.97.1:39109)
0926 03:02:29.467270 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:29.467278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:29.475817 (+ 8539us) server_negotiation.cc:407] Connection header received
0926 03:02:29.476021 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:29.476030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:29.476138 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:29.476264 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:29.478587 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.479834 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.482124 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.482751 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.482844 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.486508 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:29.486522 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:29.486539 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:29.486579 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:29.490591 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:29.491560 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:29.491568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:29.491572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:29.491762 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:29.492845 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:29.492852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:29.492859 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:29.493320 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:29.493651 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:29.494276 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 03:02:29.494515 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:02:29.498250 13780 log.cc:1196] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/64c505f37a4640f18dd6e4f8d8a09b90
I20260926 03:02:29.498641 13442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:29.499578 13781 ts_tablet_manager.cc:1939] T 64c505f37a4640f18dd6e4f8d8a09b90 P 43bd9dab85834f2aa27717a11615db95: Deleting consensus metadata
I20260926 03:02:29.502388 13237 catalog_manager.cc:4841] TS 43bd9dab85834f2aa27717a11615db95 (127.0.97.2:33379): tablet 64c505f37a4640f18dd6e4f8d8a09b90 (table test-table [id=564a0514e9dc476b892b80dd0eaec7fb]) successfully deleted
I20260926 03:02:29.589761 13780 ts_tablet_manager.cc:1939] T 64c505f37a4640f18dd6e4f8d8a09b90 P 6b7bc993eab946148a8522b50f498df2: Deleting consensus metadata
I20260926 03:02:29.590443 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13287
W20260926 03:02:29.613528 13235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39109 (error 108)
I20260926 03:02:29.614270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13451
I20260926 03:02:29.615772 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.614805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37764)
0926 03:02:29.615368 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.615534 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":572,"threads_started":1}
W20260926 03:02:29.617219 13235 catalog_manager.cc:4584] TS 6b7bc993eab946148a8522b50f498df2 (127.0.97.1:39109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64c505f37a4640f18dd6e4f8d8a09b90: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
I20260926 03:02:29.629573 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.615012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.0.97.2:33379)
0926 03:02:29.615236 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:29.615242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:29.615263 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:02:29.615436 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:29.615441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:29.615529 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:29.615637 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:29.617364 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.618101 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.619884 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.620348 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.620488 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.623163 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:29.623179 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:29.623197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:29.623235 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:29.626864 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:29.627503 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:29.627513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:29.627517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:29.627674 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:29.628009 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:29.628014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:29.628017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:29.628437 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:29.628729 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:29.628926 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 03:02:29.629163 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 03:02:29.630294 13586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:29.670846 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.670320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37772)
0926 03:02:29.670549 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.670668 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:02:29.672104 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.671637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37774)
0926 03:02:29.671825 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.671929 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:02:29.720196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13451
I20260926 03:02:29.732914 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.732343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37780)
0926 03:02:29.732627 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.732729 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 03:02:29.738157 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.733803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37788)
0926 03:02:29.737842 (+ 4039us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.737944 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3913,"threads_started":1}
I20260926 03:02:29.740675 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13595
I20260926 03:02:29.756489 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.741535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.97.3:42013)
0926 03:02:29.741720 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:29.741727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:29.741747 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:02:29.741948 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:29.741953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:29.742042 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:29.742177 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:29.744449 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.745734 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.747100 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:29.747530 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:29.747683 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:29.750198 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:29.750210 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:29.750225 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:29.750256 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:29.753622 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:29.754358 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:29.754374 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:29.754377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:29.754521 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:29.754910 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:29.754916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:29.754919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:29.755238 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:29.755547 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:29.755859 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:02:29.756064 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:02:29.757270 13720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:29.826166 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.825539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37790)
0926 03:02:29.825790 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.825948 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 03:02:29.827267 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:29.826833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37792)
0926 03:02:29.827043 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:29.827137 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 03:02:29.875372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13595
I20260926 03:02:29.894585 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13217
I20260926 03:02:29.895722 13280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:30.006054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13217
I20260926 03:02:30.006115 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:30.005483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:37808)
0926 03:02:30.005679 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:30.005791 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 03:02:30.030925 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:02:30.366604 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:02:30.366775 388 mini_ranger.cc:66] Stopping Ranger...
I20260926 03:02:30.711975 388 mini_ranger.cc:68] Stopped Ranger
2026-09-26 03:02:30.712 UTC [12269] LOG: received smart shutdown request
2026-09-26 03:02:30.715 UTC [12269] LOG: background worker "logical replication launcher" (PID 12278) exited with exit code 1
2026-09-26 03:02:30.716 UTC [12273] LOG: shutting down
2026-09-26 03:02:30.883 UTC [12269] LOG: database system is shut down
2026-09-26T03:02:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50974 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:02:31 dist-test-slave-fccw krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:02:31 dist-test-slave-fccw krb5kdc[13798](info): set up 2 sockets
Sep 26 03:02:31 dist-test-slave-fccw krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260926 03:02:32.105428 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:02:32 dist-test-slave-fccw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:02:32.316043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37551
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37551,127.0.97.61:37941,127.0.97.60:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:32.634117 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:32.634442 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:32.634622 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:32.641394 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:32.641544 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:32.641659 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:32.641757 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:32.641836 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:32.651126 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37551,127.0.97.61:37941,127.0.97.60:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:32.652292 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:32.653599 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:32.664841 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:32.664882 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:32.664917 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:32.665642 13811 server_base.cc:1034] running on GCE node
I20260926 03:02:32.666348 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:32.667562 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:32.668820 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790391752668789 us; error 38 us; skew 500 ppm
I20260926 03:02:32.669216 13811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:32 dist-test-slave-fccw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:32.672051 13811 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:02:32.673467 13811 webserver.cc:460] Webserver started at http://127.0.97.62:36185/ using document root <none> and password file <none>
I20260926 03:02:32.674103 13811 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:32.674247 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:32.674552 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:32.677364 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "4c280f54fa9e4b8eb08c26a4f3723444"
format_stamp: "Formatted at 2026-09-26 03:02:32 on dist-test-slave-fccw"
server_key: "4ca72cb91045901d0b6d34244062ac88"
server_key_iv: "7bf2b5f4a37002106bff82d8ddf9fa23"
server_key_version: "encryptionkey@0"
I20260926 03:02:32.678098 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c280f54fa9e4b8eb08c26a4f3723444"
format_stamp: "Formatted at 2026-09-26 03:02:32 on dist-test-slave-fccw"
server_key: "4ca72cb91045901d0b6d34244062ac88"
server_key_iv: "7bf2b5f4a37002106bff82d8ddf9fa23"
server_key_version: "encryptionkey@0"
I20260926 03:02:32.682787 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:02:32.685977 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:32.687139 13811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:02:32.687326 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "4c280f54fa9e4b8eb08c26a4f3723444"
format_stamp: "Formatted at 2026-09-26 03:02:32 on dist-test-slave-fccw"
server_key: "4ca72cb91045901d0b6d34244062ac88"
server_key_iv: "7bf2b5f4a37002106bff82d8ddf9fa23"
server_key_version: "encryptionkey@0"
I20260926 03:02:32.687518 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:32.705775 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:32.709404 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:32.709674 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:32.727497 13811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37551
I20260926 03:02:32.727573 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37551 every 8 connection(s)
I20260926 03:02:32.728770 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:02:32.732132 13878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:32.736709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13811
I20260926 03:02:32.736930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:02:32.737756 388 external_mini_cluster.cc:1428] Setting key 668d06933a6fba3721471e0e6a4886a2
I20260926 03:02:32.739440 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:02:32 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 03:02:32.763257 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.743158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37551 (local address 127.0.0.1:40900)
0926 03:02:32.745821 (+ 2663us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.745875 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:32.747499 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:32.748293 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:32.748299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:32.749039 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:32.749369 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:32.749386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:32.750933 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:32.750942 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:32.751973 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:32.751984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:32.752885 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:32.752891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:32.753127 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:32.753793 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:32.753819 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:32.756047 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:32.759205 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:32.759217 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:32.759227 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:32.759609 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:32.760036 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:32.760043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:32.760046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:32.760192 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:32.760886 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:32.760893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:32.761150 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 03:02:32.761377 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 03:02:32.761979 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2407,"mutex_wait_us":165,"spinlock_wait_cycles":43776,"thread_start_us":2208,"threads_started":1}
I20260926 03:02:32.763275 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.743158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.0.97.62:37551)
0926 03:02:32.745788 (+ 2630us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:32.745802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:32.746012 (+ 210us) server_negotiation.cc:407] Connection header received
0926 03:02:32.747868 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:32.747873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:32.748078 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:32.748283 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:32.749567 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:32.750765 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:32.752132 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:32.752662 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:32.752859 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:32.756249 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:32.756257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:32.756267 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:32.756296 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:32.759058 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:32.759736 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:32.759745 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:32.759753 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:32.759929 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:32.760304 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:32.760308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:32.760312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:32.760751 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:32.760953 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:32.761512 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 03:02:32.761867 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":2188,"thread_start_us":1955,"threads_started":1}
I20260926 03:02:32.766224 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:02:32.767364 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.766885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:33966)
0926 03:02:32.767087 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.767208 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 03:02:32.767668 13828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
I20260926 03:02:32.769299 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.768827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:33974)
0926 03:02:32.769031 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.769146 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:02:32.769547 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37941: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:02:32.784145 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37941
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37551,127.0.97.61:37941,127.0.97.60:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:02:32.818907 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } attempt: 1
I20260926 03:02:32.820101 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.819654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:33980)
0926 03:02:32.819872 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.819973 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:02:32.821146 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.820778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:33982)
0926 03:02:32.820912 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.820974 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:02:32.821368 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37941: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
I20260926 03:02:32.889691 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } attempt: 2
I20260926 03:02:32.890902 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.890442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:33998)
0926 03:02:32.890664 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.890757 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 03:02:32.891968 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.891664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:34006)
0926 03:02:32.891780 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.891858 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 03:02:32.892206 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37941: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
I20260926 03:02:32.983476 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } attempt: 3
I20260926 03:02:32.984834 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.984328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:34020)
0926 03:02:32.984567 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.984656 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 03:02:32.985978 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:32.985607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:34024)
0926 03:02:32.985770 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:32.985843 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 03:02:32.986227 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37941: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
W20260926 03:02:33.108640 13885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:33.108955 13885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:33.109146 13885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:33.116258 13885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:33.116394 13885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:33.116497 13885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:33.116596 13885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:33.116660 13885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:33.127362 13885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37551,127.0.97.61:37941,127.0.97.60:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:33.128706 13885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:33.129494 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } attempt: 4
I20260926 03:02:33.130294 13885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:02:33.130669 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.130128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:34028)
0926 03:02:33.130460 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.130550 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260926 03:02:33.131635 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.131292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:34044)
0926 03:02:33.131462 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.131528 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:02:33.131880 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:37941: Network error: Client connection negotiation failed: client connection to 127.0.97.61:37941: connect: Connection refused (error 111)
W20260926 03:02:33.141687 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:33.141693 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:33.141631 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:33.142782 13885 server_base.cc:1034] running on GCE node
I20260926 03:02:33.143558 13885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:33.144901 13885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:33.146130 13885 hybrid_clock.cc:648] HybridClock initialized: now 1790391753146074 us; error 49 us; skew 500 ppm
I20260926 03:02:33.146483 13885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:33.155057 13885 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:02:33.156549 13885 webserver.cc:460] Webserver started at http://127.0.97.61:43021/ using document root <none> and password file <none>
I20260926 03:02:33.157313 13885 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:33.157456 13885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:33.157764 13885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:33.160672 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data/instance:
uuid: "ac09ddfe293043dd9629b4ce002a9bc9"
format_stamp: "Formatted at 2026-09-26 03:02:33 on dist-test-slave-fccw"
server_key: "9b80bc99149e0fff88c33b316a3463ed"
server_key_iv: "81687fa287d990e36dc2c3852df701df"
server_key_version: "encryptionkey@0"
I20260926 03:02:33.161440 13885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/wal/instance:
uuid: "ac09ddfe293043dd9629b4ce002a9bc9"
format_stamp: "Formatted at 2026-09-26 03:02:33 on dist-test-slave-fccw"
server_key: "9b80bc99149e0fff88c33b316a3463ed"
server_key_iv: "81687fa287d990e36dc2c3852df701df"
server_key_version: "encryptionkey@0"
I20260926 03:02:33.166364 13885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:02:33.169561 13900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:33.170913 13885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:02:33.171103 13885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/wal
uuid: "ac09ddfe293043dd9629b4ce002a9bc9"
format_stamp: "Formatted at 2026-09-26 03:02:33 on dist-test-slave-fccw"
server_key: "9b80bc99149e0fff88c33b316a3463ed"
server_key_iv: "81687fa287d990e36dc2c3852df701df"
server_key_version: "encryptionkey@0"
I20260926 03:02:33.171299 13885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:33.186653 13885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:33.191035 13885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:33.191362 13885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:33.210062 13885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37941
I20260926 03:02:33.210094 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37941 every 8 connection(s)
I20260926 03:02:33.211436 13885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:02:33.213063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13885
I20260926 03:02:33.213250 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-1/wal/instance
I20260926 03:02:33.213781 388 external_mini_cluster.cc:1428] Setting key b1aa96b33eb425d5a2e9111b401e49c7
I20260926 03:02:33.215965 13952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:33.224357 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:33.244594 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.227048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.0.97.62:37551)
0926 03:02:33.227264 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.227270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.228387 (+ 1117us) server_negotiation.cc:407] Connection header received
0926 03:02:33.230255 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.230264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.230357 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.230489 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.231919 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.233086 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.234563 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.235084 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.235186 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.238879 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.238884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.238887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.238920 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.241612 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.242322 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.242329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.242332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.242453 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.242884 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.242890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.242893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.243162 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.243320 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.244208 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.244374 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:02:33.246274 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.227397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37551 (local address 127.0.0.1:40910)
0926 03:02:33.228183 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.228238 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.229977 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.230726 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.230734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.231208 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.231734 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.231750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.233230 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.233234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.234364 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.234387 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.235323 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.235329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.235587 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.236233 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.236291 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.238654 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.241762 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.241771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.241786 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.242199 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.242584 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.242590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.242594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.242772 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.243297 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.243304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.243807 (+ 503us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.244113 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.244654 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":354,"threads_started":1}
I20260926 03:02:33.248050 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:02:33.262295 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.248858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:34052)
0926 03:02:33.249068 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.249083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.249159 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.250185 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.250189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.250630 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.250746 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.250753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.252064 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.252069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.252864 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.252881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.253696 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.253701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.253906 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.254469 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.254484 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.256632 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.259950 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.259953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.259956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.260294 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.260710 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.260713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.260715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.260841 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.261495 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.261499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.261637 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.261806 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.261999 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:02:33.262537 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.249171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.0.97.61:37941)
0926 03:02:33.249806 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.249811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.249834 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:02:33.249871 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.249878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.250036 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.250201 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.250894 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.251919 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.253031 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.253540 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.253681 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.256870 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.256898 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.256911 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.256952 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.259825 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.260401 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.260407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.260410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.260573 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.260937 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.260940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.260942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.261382 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.261552 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.261926 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.262106 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":379,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:02:33.264823 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:02:33.265270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45909
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37551,127.0.97.61:37941,127.0.97.60:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:02:33.266162 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.265620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:60962)
0926 03:02:33.265826 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.265966 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 03:02:33.266448 13901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.268060 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.267640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:60970)
0926 03:02:33.267820 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.267919 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 03:02:33.268292 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.317667 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 1
I20260926 03:02:33.318989 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.318537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:60986)
0926 03:02:33.318735 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.318837 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 03:02:33.319972 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.319642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:60992)
0926 03:02:33.319788 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.319866 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 03:02:33.320182 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.388408 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 2
I20260926 03:02:33.389560 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.389115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55568)
0926 03:02:33.389301 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.389410 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:02:33.390601 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.390206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55580)
0926 03:02:33.390364 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.390490 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:02:33.390796 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.431161 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } attempt: 5
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:02:33.444571 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.431971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:44240)
0926 03:02:33.432219 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.432243 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.432415 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.432857 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.432863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.433298 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.433488 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.433497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.434495 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.434501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.435383 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.435402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.436127 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.436132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.436323 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.436976 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.437017 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.439232 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.442170 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.442176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.442180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.442622 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.442980 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.442985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.442988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.443142 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.443807 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.443810 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.443951 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.444061 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.444315 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 03:02:33.444736 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.432317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.0.97.61:37941)
0926 03:02:33.432542 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.432547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.432572 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:02:33.432624 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.432627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.432699 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.432812 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.433652 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.434345 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.435535 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.435979 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.436118 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.439424 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.439428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.439431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.439460 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.442029 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.442741 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.442753 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.442756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.442862 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.443279 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.443284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.443287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.443653 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.443840 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.444193 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.444413 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 03:02:33.445812 13878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:02:33.446910 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.446493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55584)
0926 03:02:33.446646 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.446729 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:02:33.447824 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.447506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55596)
0926 03:02:33.447661 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.447741 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 03:02:33.448063 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.481998 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 3
I20260926 03:02:33.483129 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.482642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55612)
0926 03:02:33.482844 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.482956 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 03:02:33.484193 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.483811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55616)
0926 03:02:33.484019 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.484095 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 03:02:33.484411 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.499361 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 1
I20260926 03:02:33.500438 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.500011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55628)
0926 03:02:33.500188 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.500265 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 03:02:33.501432 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.501149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55630)
0926 03:02:33.501275 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.501336 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:02:33.501662 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.569895 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 2
I20260926 03:02:33.571031 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.570590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55634)
0926 03:02:33.570791 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.570874 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:02:33.571998 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.571718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55650)
0926 03:02:33.571844 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.571905 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:02:33.572276 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
W20260926 03:02:33.591238 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:33.591591 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:33.591764 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:33.598734 13959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:33.598853 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:33.598938 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:33.599058 13959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:33.599146 13959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:33.608589 13959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37551,127.0.97.61:37941,127.0.97.60:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:33.609881 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:33.611195 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:33.622876 13965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:33.622891 13964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:33.622898 13967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:33.623657 13959 server_base.cc:1034] running on GCE node
I20260926 03:02:33.624424 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:33.625813 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:33.627040 13959 hybrid_clock.cc:648] HybridClock initialized: now 1790391753626991 us; error 63 us; skew 500 ppm
I20260926 03:02:33.627370 13959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:02:33.627617 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 4
I20260926 03:02:33.628762 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.628236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55664)
0926 03:02:33.628493 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.628598 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:02:33.629726 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.629392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55666)
0926 03:02:33.629565 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.629629 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:02:33.629943 13952 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:33.630811 13959 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:02:33.632143 13959 webserver.cc:460] Webserver started at http://127.0.97.60:34425/ using document root <none> and password file <none>
I20260926 03:02:33.632788 13959 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:33.632929 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:33.633231 13959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:33.636102 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data/instance:
uuid: "c86a19924dee449fa7ca4748f294eff9"
format_stamp: "Formatted at 2026-09-26 03:02:33 on dist-test-slave-fccw"
server_key: "4762725599e70aafd5d4a4b96e566944"
server_key_iv: "80119e0ba27785bf7a00bceabddd578a"
server_key_version: "encryptionkey@0"
I20260926 03:02:33.636854 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/wal/instance:
uuid: "c86a19924dee449fa7ca4748f294eff9"
format_stamp: "Formatted at 2026-09-26 03:02:33 on dist-test-slave-fccw"
server_key: "4762725599e70aafd5d4a4b96e566944"
server_key_iv: "80119e0ba27785bf7a00bceabddd578a"
server_key_version: "encryptionkey@0"
I20260926 03:02:33.641693 13959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:02:33.644901 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:33.646199 13959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:02:33.646404 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/wal
uuid: "c86a19924dee449fa7ca4748f294eff9"
format_stamp: "Formatted at 2026-09-26 03:02:33 on dist-test-slave-fccw"
server_key: "4762725599e70aafd5d4a4b96e566944"
server_key_iv: "80119e0ba27785bf7a00bceabddd578a"
server_key_version: "encryptionkey@0"
I20260926 03:02:33.646605 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:33.664301 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:33.668059 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:33.668325 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:33.678499 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 3
I20260926 03:02:33.679684 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.679173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55680)
0926 03:02:33.679435 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.679533 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:02:33.680811 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.680459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55682)
0926 03:02:33.680656 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.680721 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260926 03:02:33.681097 13878 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45909: connect: Connection refused (error 111)
I20260926 03:02:33.687659 13959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45909
I20260926 03:02:33.687695 14025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45909 every 8 connection(s)
I20260926 03:02:33.688944 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:02:33.692530 14026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:33.696254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 13959
I20260926 03:02:33.696439 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790391557890803-388-0/minicluster-data/master-2/wal/instance
I20260926 03:02:33.696949 388 external_mini_cluster.cc:1428] Setting key 6d48587fb3cd2085fffe8e93447c436e
I20260926 03:02:33.699929 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:33.719033 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.703429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.0.97.62:37551)
0926 03:02:33.703589 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.703593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.704386 (+ 793us) server_negotiation.cc:407] Connection header received
0926 03:02:33.704568 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.704574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.704646 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.704746 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.705863 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.706527 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.707921 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.708525 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.708692 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.712442 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.712468 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.712475 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.712529 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.715853 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.716630 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.716634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.716638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.716821 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.717324 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.717328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.717331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.717669 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.717830 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.718441 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.718662 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:02:33.719906 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.698718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.97.60:45909)
0926 03:02:33.699478 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.699488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.699527 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:02:33.701147 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.701175 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.701576 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.701985 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.703634 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.705331 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.706711 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.707389 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.707584 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.711938 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.711948 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.711963 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.712000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.715749 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.716629 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.716636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.716640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.716850 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.717319 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.717323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.717326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.717982 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.718181 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.718621 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.718899 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":564,"thread_start_us":388,"threads_started":1}
I20260926 03:02:33.719990 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.703419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37551 (local address 127.0.0.1:51198)
0926 03:02:33.704231 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.704275 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.704420 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.704803 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.704808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.705462 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.705714 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.705722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.706690 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.706697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.707758 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.707770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.708768 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.708777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.708941 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.709583 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.709603 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.712263 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.715984 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.715998 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.716011 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.716526 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.716981 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.716985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.716988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.717209 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.717795 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.717801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.718015 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.718305 (+ 290us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.718772 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":421,"threads_started":1}
I20260926 03:02:33.721359 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:33.735926 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.721958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37941 (local address 127.0.0.1:44246)
0926 03:02:33.722130 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.722149 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.722261 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.722587 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.722593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.723012 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.723194 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.723205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.724087 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.724090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.725552 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.725567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.726249 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.726255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.726473 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.726964 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.726985 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.729519 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.733226 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.733232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.733238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.733745 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.734219 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.734224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.734228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.734381 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.735019 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.735024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.735188 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.735329 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.735587 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:02:33.736657 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.722081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.0.97.61:37941)
0926 03:02:33.722291 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.722297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.722316 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:02:33.722375 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.722382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.722456 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.722531 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.723311 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.723962 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.725685 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.726092 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.726233 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.729714 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.729722 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.729729 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.729766 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.733085 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.733886 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.733893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.733896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.734085 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.734507 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.734514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.734518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.734827 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.734959 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.735465 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.736264 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:02:33.737920 14026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:02:33.740407 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.722962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.0.97.62:37551)
0926 03:02:33.723114 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.723118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.723158 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:02:33.724114 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.724117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.724167 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.724234 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.725873 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.726576 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.728475 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.728919 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.729077 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.734000 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.734005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.734008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.734036 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.737545 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.738282 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.738287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.738290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.738435 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.738952 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.738959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.738963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.739344 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.739624 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.739811 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.740008 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:02:33.752115 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.738800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55702)
0926 03:02:33.738950 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.738968 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.739073 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.739355 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.739359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.739814 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.739985 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.739995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.741248 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.741252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.742555 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.742571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.743272 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.743278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.743494 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.743979 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.744004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.745922 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.749594 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.749597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.749599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.750066 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.750490 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.750495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.750500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.750613 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.751131 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.751135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.751259 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.751420 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.751707 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:02:33.752141 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.738673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.97.60:45909)
0926 03:02:33.738837 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.738845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.739036 (+ 191us) server_negotiation.cc:407] Connection header received
0926 03:02:33.739173 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.739177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.739229 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.739299 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.740119 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.741091 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.742699 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.743124 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.743253 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.746098 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.746102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.746105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.746131 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.749427 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.750190 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.750198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.750203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.750376 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.750718 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.750725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.750728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.751020 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.751156 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.751525 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.751848 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":59}
I20260926 03:02:33.756716 14026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9: Bootstrap starting.
I20260926 03:02:33.757174 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.742125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.97.61:37941)
0926 03:02:33.742266 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.742270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.742284 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:02:33.742428 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.742431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.742481 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.742550 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.744009 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.744743 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.746135 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.746552 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.746681 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.751625 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.751632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.751636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.751673 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.754498 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.755203 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.755209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.755213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.755366 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.755780 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.755785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.755788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.756029 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.756313 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.756581 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.756801 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:02:33.760630 14026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:33.761772 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:02:33.761761 14026 log.cc:824] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:33.764626 14026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9: No bootstrap required, opened a new log
I20260926 03:02:33.777522 14026 raft_consensus.cc:348] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.777941 14026 raft_consensus.cc:374] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:33.778059 14026 raft_consensus.cc:729] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86a19924dee449fa7ca4748f294eff9, State: Initialized, Role: FOLLOWER
I20260926 03:02:33.778759 14026 consensus_queue.cc:260] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.785688 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:33.786113 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:02:33.788138 14026 sys_catalog.cc:564] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:33.796298 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.767669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51208 (local address 127.0.97.62:37551)
0926 03:02:33.769317 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.769326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.774513 (+ 5187us) server_negotiation.cc:407] Connection header received
0926 03:02:33.774947 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.774953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.775031 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.775101 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.777647 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.778709 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.781950 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.782856 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.782987 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.787665 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.787671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.787676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.787724 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.791830 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.793166 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.793175 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.793181 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.793411 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.793966 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.793974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.793979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.794444 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.794747 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.795459 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.795783 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1475}
I20260926 03:02:33.796749 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.767465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.97.61:37941)
0926 03:02:33.767617 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.767622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.778718 (+ 11096us) server_negotiation.cc:407] Connection header received
0926 03:02:33.778906 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.778914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.779009 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.779093 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.781955 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.783114 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.785526 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.786159 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.786262 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.789622 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.789631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.789639 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.789684 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.793338 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.794271 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.794279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.794285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.794452 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.794893 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.794899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.794902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.795320 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.795569 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.796132 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.796382 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:02:33.800625 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.767344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.97.60:45909)
0926 03:02:33.767543 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.767549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.781073 (+ 13524us) server_negotiation.cc:407] Connection header received
0926 03:02:33.781524 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.781531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.781615 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.781689 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.783821 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.785092 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.787512 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.788301 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.788406 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.793075 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.793083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.793088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.793131 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.796754 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.798117 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.798122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.798125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.798281 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.798740 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.798744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.798749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.799142 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.799424 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.799782 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.800095 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 03:02:33.808898 14050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:02:33.809062 14050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:02:33.805341 13959 master_runner.cc:186] Error getting master registration for 127.0.97.62:37551: OK, instance_id { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" instance_seqno: 1790391752722446 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:02:33.814277 13959 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:02:33.858317 13878 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 4
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:02:33.872259 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.859103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.0.97.60:45909)
0926 03:02:33.859304 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.859310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.859326 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:02:33.859371 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.859376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.859480 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.859618 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.860356 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.861373 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.862607 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.863016 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.863165 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.866391 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.866396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.866399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.866425 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.869722 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.870498 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.870502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.870505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.870638 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.871033 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.871036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.871038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.871285 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.871388 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.871806 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.871972 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:02:33.872464 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.858867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55730)
0926 03:02:33.859053 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.859085 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.859209 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.859639 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.859643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.860048 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.860212 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.860221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.861513 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.861520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.862436 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.862446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.863166 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.863172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.863359 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.864023 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.864039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.866216 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.869865 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.869870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.869875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.870381 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.870750 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.870755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.870759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.870929 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.871406 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.871412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.871577 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.871735 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.872047 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:02:33.877043 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: Bootstrap starting.
I20260926 03:02:33.880156 13878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:33.881515 13878 log.cc:824] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:33.884027 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: No bootstrap required, opened a new log
I20260926 03:02:33.891144 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.891477 13878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:33.891613 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c280f54fa9e4b8eb08c26a4f3723444, State: Initialized, Role: FOLLOWER
I20260926 03:02:33.892179 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.893082 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:33.893456 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:02:33.894375 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:02:33.904381 14064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:02:33.904510 14064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:02:33.910059 14039 raft_consensus.cc:482] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:33.910346 14039 raft_consensus.cc:504] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.911939 14039 leader_election.cc:288] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c280f54fa9e4b8eb08c26a4f3723444 (127.0.97.62:37551), ac09ddfe293043dd9629b4ce002a9bc9 (127.0.97.61:37941)
I20260926 03:02:33.913308 13853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c86a19924dee449fa7ca4748f294eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" is_pre_election: true
I20260926 03:02:33.913872 13853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a19924dee449fa7ca4748f294eff9 in term 0.
I20260926 03:02:33.914551 13977 leader_election.cc:302] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c280f54fa9e4b8eb08c26a4f3723444, c86a19924dee449fa7ca4748f294eff9; no voters:
I20260926 03:02:33.914973 14039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:02:33.915105 14039 raft_consensus.cc:482] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:02:33.915205 14039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:33.918494 14039 raft_consensus.cc:504] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.919556 13853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c86a19924dee449fa7ca4748f294eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c280f54fa9e4b8eb08c26a4f3723444"
I20260926 03:02:33.919755 13853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [term 0 FOLLOWER]: Advancing to term 1
W20260926 03:02:33.920495 13980 tablet.cc:2369] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:02:33.903798 13811 master_runner.cc:186] Error getting master registration for 127.0.97.61:37941: OK, instance_id { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" instance_seqno: 1790391753204724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:02:33.922405 13811 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:02:33.923089 14039 leader_election.cc:288] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [CANDIDATE]: Term 1 election: Requested vote from peers 4c280f54fa9e4b8eb08c26a4f3723444 (127.0.97.62:37551), ac09ddfe293043dd9629b4ce002a9bc9 (127.0.97.61:37941)
I20260926 03:02:33.923434 13853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a19924dee449fa7ca4748f294eff9 in term 1.
I20260926 03:02:33.923839 13977 leader_election.cc:302] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c280f54fa9e4b8eb08c26a4f3723444, c86a19924dee449fa7ca4748f294eff9; no voters:
I20260926 03:02:33.924152 14039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:02:33.924783 14039 raft_consensus.cc:686] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [term 1 LEADER]: Becoming Leader. State: Replica: c86a19924dee449fa7ca4748f294eff9, State: Running, Role: LEADER
I20260926 03:02:33.925408 14039 consensus_queue.cc:237] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.929132 13952 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } attempt: 5
Sep 26 03:02:33 dist-test-slave-fccw krb5kdc[13798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:02:33.955646 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.929652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45909 (local address 127.0.0.1:55742)
0926 03:02:33.930017 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:33.930040 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:33.930146 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:33.931141 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:33.931147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:33.931696 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:33.931908 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.931921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.933580 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.933588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.935054 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:33.935070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.936396 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.936403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:33.936599 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.937806 (+ 1207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:33.937862 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:33.942172 (+ 4310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:33.949791 (+ 7619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.949824 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.949829 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.950873 (+ 1044us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.951772 (+ 899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:33.951778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:33.951810 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:33.952089 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:33.953606 (+ 1517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:33.953611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:33.953915 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 03:02:33.954139 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 03:02:33.954659 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260926 03:02:33.957818 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:33.930462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.0.97.60:45909)
0926 03:02:33.930647 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:33.930653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:33.930670 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:02:33.930720 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:33.930726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:33.930798 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:33.930892 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:33.932114 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.933350 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.935265 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:33.936000 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:33.936125 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:33.942489 (+ 6364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:33.942496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:33.942500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:33.942540 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:33.949519 (+ 6979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.951190 (+ 1671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.951196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.951200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.951547 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:33.952356 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:33.952361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:33.952365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:33.953234 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:33.955716 (+ 2482us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:33.956462 (+ 746us) server_negotiation.cc:299] Negotiation successful
0926 03:02:33.957071 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:02:33.962343 13832 tablet.cc:2369] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:02:33.964783 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9: Bootstrap starting.
I20260926 03:02:33.972581 13952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:33.974992 13952 log.cc:824] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:33.978691 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86a19924dee449fa7ca4748f294eff9. Latest consensus state: current_term: 1 leader_uuid: "c86a19924dee449fa7ca4748f294eff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:33.979146 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:33.980618 14069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:02:33.984978 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9: No bootstrap required, opened a new log
I20260926 03:02:33.985397 14069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:02:33.997562 13952 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:33.997979 13952 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:33.998095 13952 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac09ddfe293043dd9629b4ce002a9bc9, State: Initialized, Role: FOLLOWER
I20260926 03:02:33.998802 13952 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } }
I20260926 03:02:34.005771 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:34.006152 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:02:34.007937 13952 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:34.007997 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c86a19924dee449fa7ca4748f294eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac09ddfe293043dd9629b4ce002a9bc9"
I20260926 03:02:34.008471 13926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:34.011909 13926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a19924dee449fa7ca4748f294eff9 in term 1.
I20260926 03:02:34.012367 13927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c86a19924dee449fa7ca4748f294eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" is_pre_election: true
I20260926 03:02:34.012714 13927 raft_consensus.cc:2362] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c86a19924dee449fa7ca4748f294eff9 in term 1. Re-sending same reply.
I20260926 03:02:34.017943 13926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [term 1 FOLLOWER]: Refusing update from remote peer c86a19924dee449fa7ca4748f294eff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:34.018865 14066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:34.019706 13853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [term 1 FOLLOWER]: Refusing update from remote peer c86a19924dee449fa7ca4748f294eff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:34.020524 14066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:34.046231 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86a19924dee449fa7ca4748f294eff9. Latest consensus state: current_term: 1 leader_uuid: "c86a19924dee449fa7ca4748f294eff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:34.046571 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:02:34.047578 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86a19924dee449fa7ca4748f294eff9. Latest consensus state: current_term: 1 leader_uuid: "c86a19924dee449fa7ca4748f294eff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:34.047847 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:02:34.061563 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c86a19924dee449fa7ca4748f294eff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:34.061811 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:02:34.065871 14053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c86a19924dee449fa7ca4748f294eff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:34.066124 14053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:02:34.069998 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c86a19924dee449fa7ca4748f294eff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:34.070365 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:34.070629 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c86a19924dee449fa7ca4748f294eff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c280f54fa9e4b8eb08c26a4f3723444" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37551 } } peers { permanent_uuid: "ac09ddfe293043dd9629b4ce002a9bc9" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37941 } } peers { permanent_uuid: "c86a19924dee449fa7ca4748f294eff9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45909 } } }
I20260926 03:02:34.070895 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:34.071558 14069 catalog_manager.cc:1347] Generated new cluster ID: d2ac5fb710984c14b7077676aa092df6
I20260926 03:02:34.071660 14069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:02:34.142307 14069 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:02:34.144021 14089 catalog_manager.cc:1259] Loaded cluster ID: d2ac5fb710984c14b7077676aa092df6
I20260926 03:02:34.144130 14089 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9: loading cluster ID for follower catalog manager: success
I20260926 03:02:34.145227 14069 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:02:34.147598 14089 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9: acquiring CA information for follower catalog manager: success
W20260926 03:02:34.154340 14089 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260926 03:02:34.159476 14069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9: Generated new TSK 0
I20260926 03:02:34.160421 14069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:02:34.906680 14064 catalog_manager.cc:1259] Loaded cluster ID: d2ac5fb710984c14b7077676aa092df6
I20260926 03:02:34.906778 14064 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: loading cluster ID for follower catalog manager: success
I20260926 03:02:34.909807 14064 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: acquiring CA information for follower catalog manager: success
I20260926 03:02:34.911664 14064 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c280f54fa9e4b8eb08c26a4f3723444: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:02:35.156916 14089 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac09ddfe293043dd9629b4ce002a9bc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:02:35.319247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13811
I20260926 03:02:35.320400 13873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:35.432116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13811
I20260926 03:02:35.460520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13885
I20260926 03:02:35.461804 13947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:35.547163 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:35.546390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37551 (local address 127.0.0.1:51220)
0926 03:02:35.546869 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:35.546988 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":272,"threads_started":1}
W20260926 03:02:35.547454 13977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37551: connect: Connection refused (error 111)
I20260926 03:02:35.551759 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:35.551369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37551 (local address 127.0.0.1:51226)
0926 03:02:35.551533 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:35.551635 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 03:02:35.552017 13977 consensus_peers.cc:487] T 00000000000000000000000000000000 P c86a19924dee449fa7ca4748f294eff9 -> Peer 4c280f54fa9e4b8eb08c26a4f3723444 (127.0.97.62:37551): Couldn't send request to peer 4c280f54fa9e4b8eb08c26a4f3723444. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37551: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:02:35.575471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13885
I20260926 03:02:35.595674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 13959
I20260926 03:02:35.596803 14021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:35.702906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 13959
2026-09-26T03:02:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4698 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:02:35 dist-test-slave-fccw krb5kdc[14101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:02:35 dist-test-slave-fccw krb5kdc[14101](info): set up 2 sockets
Sep 26 03:02:35 dist-test-slave-fccw krb5kdc[14101](info): commencing operation
krb5kdc: starting...
W20260926 03:02:36.789588 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:02:36 dist-test-slave-fccw krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:02:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:02:37.008338 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43099
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:37.326048 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:37.326381 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:37.326550 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:37.333451 14114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:37.333590 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:37.333695 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:37.333801 14114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:37.333869 14114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:37.343128 14114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:37.344347 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:37.345644 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:37.357018 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:37.357062 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:37.357390 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:37.357913 14114 server_base.cc:1034] running on GCE node
I20260926 03:02:37.358665 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:37.359966 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:37.361145 14114 hybrid_clock.cc:648] HybridClock initialized: now 1790391757361112 us; error 31 us; skew 500 ppm
I20260926 03:02:37.361482 14114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:37 dist-test-slave-fccw krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:37.364498 14114 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:02:37.365893 14114 webserver.cc:460] Webserver started at http://127.0.97.62:42811/ using document root <none> and password file <none>
I20260926 03:02:37.366503 14114 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:37.366596 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:37.366846 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:37.369679 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "fa5fc705367a42cb95bfb901ba1df8d6"
format_stamp: "Formatted at 2026-09-26 03:02:37 on dist-test-slave-fccw"
server_key: "c423d12d673197a185a59fcadc18e32c"
server_key_iv: "92d95cd7b47407a9ec26e4910c3e33ba"
server_key_version: "encryptionkey@0"
I20260926 03:02:37.370352 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "fa5fc705367a42cb95bfb901ba1df8d6"
format_stamp: "Formatted at 2026-09-26 03:02:37 on dist-test-slave-fccw"
server_key: "c423d12d673197a185a59fcadc18e32c"
server_key_iv: "92d95cd7b47407a9ec26e4910c3e33ba"
server_key_version: "encryptionkey@0"
I20260926 03:02:37.375144 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 03:02:37.378290 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:37.379565 14114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:02:37.379741 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "fa5fc705367a42cb95bfb901ba1df8d6"
format_stamp: "Formatted at 2026-09-26 03:02:37 on dist-test-slave-fccw"
server_key: "c423d12d673197a185a59fcadc18e32c"
server_key_iv: "92d95cd7b47407a9ec26e4910c3e33ba"
server_key_version: "encryptionkey@0"
I20260926 03:02:37.379940 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:37.393699 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:37.397269 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:37.397544 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:37.415334 14114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43099
I20260926 03:02:37.415356 14180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43099 every 8 connection(s)
I20260926 03:02:37.416685 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:02:37.418252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14114
I20260926 03:02:37.418458 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:02:37.419297 388 external_mini_cluster.cc:1428] Setting key ee09fb074d1bbd8baf8fb5e0f632c906
I20260926 03:02:37.421384 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:02:37 dist-test-slave-fccw krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:37.434398 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6: Bootstrap starting.
I20260926 03:02:37.438504 14181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:37.439709 14181 log.cc:824] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:37.442812 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:37.421138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.0.97.62:43099)
0926 03:02:37.421911 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:37.421921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:37.421960 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:02:37.423544 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:37.423564 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:37.423943 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:37.424266 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:37.426073 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:37.427358 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:37.429151 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:37.429775 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:37.429893 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:37.434537 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:37.434548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:37.434565 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:37.434609 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:37.438377 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:37.439099 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:37.439107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:37.439114 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:37.439316 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:37.439681 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:37.439687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:37.439692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:37.440368 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:37.440615 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:37.441178 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 03:02:37.441569 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":355,"threads_started":1}
I20260926 03:02:37.442922 14181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6: No bootstrap required, opened a new log
I20260926 03:02:37.446460 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:02:37.450912 14181 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43099 } }
I20260926 03:02:37.451254 14181 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:37.451378 14181 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5fc705367a42cb95bfb901ba1df8d6, State: Initialized, Role: FOLLOWER
I20260926 03:02:37.451905 14181 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43099 } }
I20260926 03:02:37.452123 14181 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:02:37.452250 14181 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:02:37.452431 14181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:37.454545 14181 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43099 } }
I20260926 03:02:37.455058 14181 leader_election.cc:302] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5fc705367a42cb95bfb901ba1df8d6; no voters:
I20260926 03:02:37.455600 14181 leader_election.cc:288] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:02:37.455731 14186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:02:37.456436 14186 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [term 1 LEADER]: Becoming Leader. State: Replica: fa5fc705367a42cb95bfb901ba1df8d6, State: Running, Role: LEADER
I20260926 03:02:37.456904 14186 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43099 } }
I20260926 03:02:37.457681 14181 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:37.459607 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43099 } } }
I20260926 03:02:37.460054 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:37.460789 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5fc705367a42cb95bfb901ba1df8d6. Latest consensus state: current_term: 1 leader_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5fc705367a42cb95bfb901ba1df8d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43099 } } }
I20260926 03:02:37.461133 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:37.462044 14193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:02:37.466723 14193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:02:37.474967 14193 catalog_manager.cc:1347] Generated new cluster ID: 0546d7105f43499a98c94a876f8f5b9f
I20260926 03:02:37.475069 14193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:02:37.520777 14193 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:02:37.522645 14193 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:02:37.532117 14193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5fc705367a42cb95bfb901ba1df8d6: Generated new TSK 0
I20260926 03:02:37.533121 14193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:02:37.558715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14114
I20260926 03:02:37.559911 14176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:37.654095 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14114
2026-09-26T03:02:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1947 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:02:37 dist-test-slave-fccw krb5kdc[14209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:02:37 dist-test-slave-fccw krb5kdc[14209](info): set up 2 sockets
Sep 26 03:02:37 dist-test-slave-fccw krb5kdc[14209](info): commencing operation
krb5kdc: starting...
W20260926 03:02:38.733477 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:02:38 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:02:38.954653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39161
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:39.278954 14222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:39.279224 14222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:39.279402 14222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:39.286588 14222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:39.286716 14222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:39.286816 14222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:39.286911 14222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:39.286995 14222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:39.296774 14222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:39.298092 14222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:39.299371 14222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:39.311525 14230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:39.311615 14228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:39.311589 14227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:39.312458 14222 server_base.cc:1034] running on GCE node
I20260926 03:02:39.313444 14222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:39.314890 14222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:39.316097 14222 hybrid_clock.cc:648] HybridClock initialized: now 1790391759316051 us; error 31 us; skew 500 ppm
I20260926 03:02:39.316480 14222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:39 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:39.323607 14222 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:02:39.325239 14222 webserver.cc:460] Webserver started at http://127.0.97.62:40689/ using document root <none> and password file <none>
I20260926 03:02:39.325958 14222 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:39.326066 14222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:39.326323 14222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:39.329495 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "1348d1e584534fb0bc12358a2d807247"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "12c587fff2e63b120b7e5a3b63db080c"
server_key_iv: "dca6c744a1dd5ace7098b4163c7e66e3"
server_key_version: "encryptionkey@0"
I20260926 03:02:39.330201 14222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "1348d1e584534fb0bc12358a2d807247"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "12c587fff2e63b120b7e5a3b63db080c"
server_key_iv: "dca6c744a1dd5ace7098b4163c7e66e3"
server_key_version: "encryptionkey@0"
I20260926 03:02:39.335135 14222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:02:39.338763 14237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:39.340077 14222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:02:39.340257 14222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "1348d1e584534fb0bc12358a2d807247"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "12c587fff2e63b120b7e5a3b63db080c"
server_key_iv: "dca6c744a1dd5ace7098b4163c7e66e3"
server_key_version: "encryptionkey@0"
I20260926 03:02:39.340464 14222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:39.362035 14222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:39.366719 14222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:39.367020 14222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:39.386153 14222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39161
I20260926 03:02:39.386176 14288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39161 every 8 connection(s)
I20260926 03:02:39.387573 14222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:02:39.391121 14289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:39.395406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14222
I20260926 03:02:39.395619 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:02:39.396420 388 external_mini_cluster.cc:1428] Setting key 38efadd5d8cc11382154701149f12226
I20260926 03:02:39.404062 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Bootstrap starting.
I20260926 03:02:39.407706 14289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:39.408975 14289 log.cc:824] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:39.411911 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: No bootstrap required, opened a new log
Sep 26 03:02:39 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:39.422550 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:39.398663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.0.97.62:39161)
0926 03:02:39.399599 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:39.399617 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:39.399671 (+ 54us) server_negotiation.cc:407] Connection header received
0926 03:02:39.402310 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:39.402338 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:39.402814 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:39.403264 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:39.404594 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:39.406730 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:39.408074 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:39.409162 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:39.409339 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:39.413759 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:39.413774 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:39.413797 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:39.413841 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:39.418080 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:39.418776 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:39.418789 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:39.418798 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:39.419000 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:39.419332 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:39.419339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:39.419343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:39.420052 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:39.420352 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:39.420648 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 03:02:39.421146 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":383,"threads_started":1}
I20260926 03:02:39.423051 14289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:39.423480 14289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:39.423599 14289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1348d1e584534fb0bc12358a2d807247, State: Initialized, Role: FOLLOWER
I20260926 03:02:39.424202 14289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:39.424440 14289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:02:39.424605 14289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:02:39.424784 14289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:39.427210 14289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:39.427692 14289 leader_election.cc:302] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1348d1e584534fb0bc12358a2d807247; no voters:
I20260926 03:02:39.428248 14289 leader_election.cc:288] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:02:39.428392 14294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:02:39.429191 14294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 LEADER]: Becoming Leader. State: Replica: 1348d1e584534fb0bc12358a2d807247, State: Running, Role: LEADER
I20260926 03:02:39.429706 14294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:39.430330 14289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:39.432798 14296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1348d1e584534fb0bc12358a2d807247. Latest consensus state: current_term: 1 leader_uuid: "1348d1e584534fb0bc12358a2d807247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } } }
I20260926 03:02:39.433246 14296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:39.433568 14295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1348d1e584534fb0bc12358a2d807247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } } }
I20260926 03:02:39.433962 14295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:39.435861 14303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:02:39.440721 14303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:02:39.449611 14303 catalog_manager.cc:1347] Generated new cluster ID: 3834cc9eb8644a4c8b3f52f4b4249c1a
I20260926 03:02:39.449723 14303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:02:39.481542 14303 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:02:39.483042 14303 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:02:39.488576 14303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Generated new TSK 0
I20260926 03:02:39.489383 14303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:02:39.540942 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39161
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:39.854907 14315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:39.855157 14315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:39.855324 14315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:39.862318 14315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:39.862418 14315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:39.862583 14315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:02:39.871671 14315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:39.872862 14315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:39.874110 14315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:39.886291 14320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:39.886404 14323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:39.886416 14321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:39.887351 14315 server_base.cc:1034] running on GCE node
I20260926 03:02:39.888020 14315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:39.888964 14315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:39.890228 14315 hybrid_clock.cc:648] HybridClock initialized: now 1790391759890196 us; error 37 us; skew 500 ppm
I20260926 03:02:39.890563 14315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:39 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:39.893662 14315 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:02:39.894923 14315 webserver.cc:460] Webserver started at http://127.0.97.1:42581/ using document root <none> and password file <none>
I20260926 03:02:39.895545 14315 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:39.895686 14315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:39.895967 14315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:39.898718 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "e0880de7f65149e3ac55a9d38583556f"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "a0078a41db700e4eabb1e425c313b3d2"
server_key_iv: "cd54f23c823250029223f1ca87b7faa2"
server_key_version: "encryptionkey@0"
I20260926 03:02:39.899462 14315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e0880de7f65149e3ac55a9d38583556f"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "a0078a41db700e4eabb1e425c313b3d2"
server_key_iv: "cd54f23c823250029223f1ca87b7faa2"
server_key_version: "encryptionkey@0"
I20260926 03:02:39.904188 14315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:02:39.907438 14330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:39.908634 14315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:02:39.908797 14315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "e0880de7f65149e3ac55a9d38583556f"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "a0078a41db700e4eabb1e425c313b3d2"
server_key_iv: "cd54f23c823250029223f1ca87b7faa2"
server_key_version: "encryptionkey@0"
I20260926 03:02:39.908983 14315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:39.941356 14315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:39.945246 14315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:39.945573 14315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:39.946496 14315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:39.947815 14315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:39.947924 14315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:39.948096 14315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:39.948165 14315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:39.979970 14315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46139
I20260926 03:02:39.980013 14442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46139 every 8 connection(s)
I20260926 03:02:39.981217 14315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:02:39.982510 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14315
I20260926 03:02:39.982741 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:02:39.983387 388 external_mini_cluster.cc:1428] Setting key 8a2da06bf15a2464819bce0fe93999f8
Sep 26 03:02:39 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:40.000932 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:39.983671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52337 (local address 127.0.97.62:39161)
0926 03:02:39.984155 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:39.984162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:39.985231 (+ 1069us) server_negotiation.cc:407] Connection header received
0926 03:02:39.987165 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:39.987168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:39.987228 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:39.987340 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:39.988833 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:39.989614 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:39.991271 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:39.991675 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:39.991807 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:39.995430 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:39.995436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:39.995439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:39.995474 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:39.997959 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:39.998679 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:39.998684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:39.998693 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:39.998807 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:39.999226 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:39.999232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:39.999235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:39.999505 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:39.999655 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:40.000334 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 03:02:40.000561 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":215,"threads_started":1}
I20260926 03:02:40.001917 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:39.984261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:52337)
0926 03:02:39.985075 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:39.985119 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:39.986864 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:39.987579 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:39.987587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:39.988096 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:39.988678 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:39.988693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:39.989761 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:39.989767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:39.991109 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:39.991121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:39.991898 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:39.991904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:39.992250 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:39.992982 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:39.993042 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:39.995214 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:39.998095 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:39.998104 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:39.998119 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:39.998549 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:39.998911 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:39.998917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:39.998920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:39.999114 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:39.999637 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:39.999644 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:39.999959 (+ 315us) client_negotiation.cc:769] Sending connection context
0926 03:02:40.000232 (+ 273us) client_negotiation.cc:240] Negotiation successful
0926 03:02:40.000723 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":347,"threads_started":1}
I20260926 03:02:40.003439 14443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39161
I20260926 03:02:40.003743 14443 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:40.004441 14443 heartbeater.cc:511] Master 127.0.97.62:39161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260926 03:02:40.006572 14254 ts_manager.cc:194] Registered new tserver with Master: e0880de7f65149e3ac55a9d38583556f (127.0.97.1:46139)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260926 03:02:40.008071 14254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:02:40.032780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39161
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:40.347632 14449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:40.347896 14449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:40.347987 14449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:40.354992 14449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:40.355086 14449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:40.355190 14449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:02:40.364692 14449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:40.365943 14449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:40.367151 14449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:40.380236 14454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:40.380299 14457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:40.380265 14455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:40.381165 14449 server_base.cc:1034] running on GCE node
I20260926 03:02:40.381740 14449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:40.382645 14449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:40.383865 14449 hybrid_clock.cc:648] HybridClock initialized: now 1790391760383806 us; error 55 us; skew 500 ppm
I20260926 03:02:40.384202 14449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:40 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:40.387475 14449 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:02:40.388918 14449 webserver.cc:460] Webserver started at http://127.0.97.2:37077/ using document root <none> and password file <none>
I20260926 03:02:40.389621 14449 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:40.389765 14449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:40.390105 14449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:40.392953 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "fbb946f8ffcb452394eb6cff78ccf728"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "6feb70b1e845d2109a39f34556abeadc"
server_key_iv: "a888605fe753471c7732b79385868fd3"
server_key_version: "encryptionkey@0"
I20260926 03:02:40.393774 14449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fbb946f8ffcb452394eb6cff78ccf728"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "6feb70b1e845d2109a39f34556abeadc"
server_key_iv: "a888605fe753471c7732b79385868fd3"
server_key_version: "encryptionkey@0"
I20260926 03:02:40.398757 14449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:02:40.402218 14464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:40.403463 14449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:02:40.403640 14449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "fbb946f8ffcb452394eb6cff78ccf728"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "6feb70b1e845d2109a39f34556abeadc"
server_key_iv: "a888605fe753471c7732b79385868fd3"
server_key_version: "encryptionkey@0"
I20260926 03:02:40.403841 14449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:40.439285 14449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:40.443382 14449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:40.443725 14449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:40.444813 14449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:40.446182 14449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:40.446286 14449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:40.446416 14449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:40.446501 14449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:40.479359 14449 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41303
I20260926 03:02:40.479413 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41303 every 8 connection(s)
I20260926 03:02:40.480651 14449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:02:40.483148 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14449
I20260926 03:02:40.483335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:02:40.483770 388 external_mini_cluster.cc:1428] Setting key 45c15a9bc26ff83ab013d96f7c81c0f6
Sep 26 03:02:40 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:40.501044 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:40.483250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35973 (local address 127.0.97.62:39161)
0926 03:02:40.483465 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:40.483471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:40.484746 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 03:02:40.487100 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:40.487104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:40.487167 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:40.487240 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:40.489357 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.490077 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.491582 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.492013 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.492150 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:40.495518 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:40.495523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:40.495526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:40.495555 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:40.498119 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:40.498823 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:40.498830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:40.498838 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:40.498952 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:40.499382 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:40.499387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:40.499392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:40.499660 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:40.499810 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:40.500449 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 03:02:40.500691 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:02:40.501909 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:40.483740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:35973)
0926 03:02:40.484553 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:40.484595 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:40.486812 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:40.487521 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:40.487536 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:40.488317 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:40.489175 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:40.489204 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.490220 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.490227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:40.491433 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:40.491445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.492253 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.492261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:40.492480 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:40.493166 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:40.493193 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:40.495302 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:40.498249 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:40.498257 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:40.498272 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:40.498690 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:40.499054 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:40.499060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:40.499063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:40.499245 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:40.499783 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:40.499790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:40.500111 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 03:02:40.500365 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:02:40.500861 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":340,"threads_started":1}
I20260926 03:02:40.503314 14577 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39161
I20260926 03:02:40.503608 14577 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:40.504285 14577 heartbeater.cc:511] Master 127.0.97.62:39161 requested a full tablet report, sending...
I20260926 03:02:40.505930 14254 ts_manager.cc:194] Registered new tserver with Master: fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
I20260926 03:02:40.506752 14254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35973
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:02:40.533857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39161
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:40.844913 14582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:40.845227 14582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:40.845325 14582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:40.852102 14582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:40.852190 14582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:40.852293 14582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:02:40.860831 14582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:40.861974 14582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:40.863255 14582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:40.875415 14587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:40.875376 14588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:40.875433 14590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:40.876161 14582 server_base.cc:1034] running on GCE node
I20260926 03:02:40.876729 14582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:40.877578 14582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:40.878764 14582 hybrid_clock.cc:648] HybridClock initialized: now 1790391760878717 us; error 49 us; skew 500 ppm
I20260926 03:02:40.879055 14582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:40 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:40.882100 14582 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:02:40.883517 14582 webserver.cc:460] Webserver started at http://127.0.97.3:37141/ using document root <none> and password file <none>
I20260926 03:02:40.884137 14582 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:40.884265 14582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:40.884559 14582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:02:40.887296 14582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "9b912c59fa722a6e729c57e9c6028a74"
server_key_iv: "76d5cbee2b1f5964b74aedafc13bf3f4"
server_key_version: "encryptionkey@0"
I20260926 03:02:40.888010 14582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "9b912c59fa722a6e729c57e9c6028a74"
server_key_iv: "76d5cbee2b1f5964b74aedafc13bf3f4"
server_key_version: "encryptionkey@0"
I20260926 03:02:40.892689 14582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260926 03:02:40.895798 14597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:40.896991 14582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:02:40.897245 14582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "9b912c59fa722a6e729c57e9c6028a74"
server_key_iv: "76d5cbee2b1f5964b74aedafc13bf3f4"
server_key_version: "encryptionkey@0"
I20260926 03:02:40.897449 14582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:40.917816 14582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:40.922114 14582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:40.922427 14582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:40.923349 14582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:40.924602 14582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:40.924705 14582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:40.924851 14582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:40.924923 14582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:40.957664 14582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46881
I20260926 03:02:40.957708 14709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46881 every 8 connection(s)
I20260926 03:02:40.958820 14582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:02:40.962958 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14582
I20260926 03:02:40.963146 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:02:40.963562 388 external_mini_cluster.cc:1428] Setting key b1bb0673d058004458b67dc3ec28a05e
Sep 26 03:02:40 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:40.977787 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:40.961225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40383 (local address 127.0.97.62:39161)
0926 03:02:40.961397 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:40.961400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:40.962525 (+ 1125us) server_negotiation.cc:407] Connection header received
0926 03:02:40.964601 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:40.964605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:40.964709 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:40.964805 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:40.966748 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.967488 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.968910 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.969352 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.969447 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:40.972498 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:40.972502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:40.972505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:40.972533 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:40.975117 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:40.975710 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:40.975713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:40.975715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:40.975835 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:40.976177 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:40.976180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:40.976182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:40.976404 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:40.976525 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:40.977228 (+ 703us) server_negotiation.cc:299] Negotiation successful
0926 03:02:40.977427 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:02:40.978798 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:40.961670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:40383)
0926 03:02:40.962347 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:40.962389 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:40.964327 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:40.965069 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:40.965080 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:40.965815 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:40.966591 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:40.966610 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.967605 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.967608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:40.968796 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:40.968807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.969529 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.969533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:40.969718 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:40.970294 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:40.970318 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:40.972335 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:40.975228 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:40.975234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:40.975244 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:40.975616 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:40.975924 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:40.975927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:40.975929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:40.976089 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:40.976523 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:40.976528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:40.976806 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 03:02:40.977094 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 03:02:40.977590 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260926 03:02:40.980108 14710 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39161
I20260926 03:02:40.980417 14710 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:40.981232 14710 heartbeater.cc:511] Master 127.0.97.62:39161 requested a full tablet report, sending...
I20260926 03:02:40.982676 14254 ts_manager.cc:194] Registered new tserver with Master: fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881)
I20260926 03:02:40.983836 14254 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40383
I20260926 03:02:40.990530 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:02:41.006759 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:40.994712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.0.97.62:39161)
0926 03:02:40.994860 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:40.994864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:40.995365 (+ 501us) server_negotiation.cc:407] Connection header received
0926 03:02:40.995485 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:40.995488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:40.995538 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:40.995616 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:40.996929 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.997647 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.998972 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:40.999377 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:40.999463 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.001765 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:41.001770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:41.001773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:41.001800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:41.004112 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.004712 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.004719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.004722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.004843 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.005280 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.005287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.005290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.005592 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:41.005862 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.006174 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.006387 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:02:41.010743 14443 heartbeater.cc:503] Master 127.0.97.62:39161 was elected leader, sending a full tablet report...
I20260926 03:02:41.010437 14254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:02:41.017778 14254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:02:41.054311 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.029125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.0.97.1:46139)
0926 03:02:41.044790 (+ 15665us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.044797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.044820 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:02:41.044895 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.044901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.045210 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.045493 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:41.046411 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.047425 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.051830 (+ 4405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.053178 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.053310 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.053544 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.053802 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.053961 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15506,"thread_start_us":15351,"threads_started":1}
I20260926 03:02:41.054385 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.028792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:33532)
0926 03:02:41.041887 (+ 13095us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:41.041945 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:41.042169 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:41.045784 (+ 3615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:41.045794 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:41.045822 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:41.046240 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.046251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.047558 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.047564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.051648 (+ 4084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.051662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.053356 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.053362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.053556 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.053624 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 03:02:41.053769 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 03:02:41.053949 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12765,"mutex_wait_us":1,"thread_start_us":15290,"threads_started":1}
I20260926 03:02:41.054505 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.028933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46881 (local address 127.0.0.1:60406)
0926 03:02:41.043834 (+ 14901us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:41.043863 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:41.044008 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:41.046025 (+ 2017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:41.046033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:41.046070 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:41.046582 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.046597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.049421 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.049430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.051838 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.051851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.053606 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.053613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.053832 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.053866 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:02:41.053927 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 03:02:41.054054 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14731,"mutex_wait_us":30,"thread_start_us":13447,"threads_started":1}
I20260926 03:02:41.055464 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.029287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.97.3:46881)
0926 03:02:41.045056 (+ 15769us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.045069 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.045112 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:02:41.045234 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.045245 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.045542 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.046815 (+ 1273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:41.047160 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.049221 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.051994 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.053382 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.054748 (+ 1366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.054973 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.055029 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.055167 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15526,"thread_start_us":16430,"threads_started":1}
I20260926 03:02:41.058457 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.047721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41303 (local address 127.0.0.1:48258)
0926 03:02:41.051313 (+ 3592us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:41.051333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:41.051448 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:41.052123 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:41.052127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:41.052153 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:41.052495 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.052507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.054383 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.054390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.056239 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.056254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.057644 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.057651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.057815 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.057846 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:02:41.057974 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 03:02:41.058095 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3465,"thread_start_us":3289,"threads_started":1}
I20260926 03:02:41.059077 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.043645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.0.97.2:41303)
0926 03:02:41.050976 (+ 7331us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.050984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.051570 (+ 586us) server_negotiation.cc:407] Connection header received
0926 03:02:41.051677 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.051684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.051918 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.052122 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:41.052682 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.054228 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.056497 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.057448 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.057591 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.057780 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.058682 (+ 902us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.058822 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7150,"thread_start_us":7025,"threads_started":1}
I20260926 03:02:41.058518 14378 tablet_service.cc:1467] Processing CreateTablet for tablet a60cfb17161841a3b1ab14142dd44903 (DEFAULT_TABLE table=test-table [id=dbe5cccf7326448494f9dcff37c38ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:41.058969 14645 tablet_service.cc:1467] Processing CreateTablet for tablet a60cfb17161841a3b1ab14142dd44903 (DEFAULT_TABLE table=test-table [id=dbe5cccf7326448494f9dcff37c38ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:41.060001 14378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60cfb17161841a3b1ab14142dd44903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:41.060266 14645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60cfb17161841a3b1ab14142dd44903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:41.063800 14512 tablet_service.cc:1467] Processing CreateTablet for tablet a60cfb17161841a3b1ab14142dd44903 (DEFAULT_TABLE table=test-table [id=dbe5cccf7326448494f9dcff37c38ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:41.064824 14512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60cfb17161841a3b1ab14142dd44903. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:41.074633 14728 tablet_bootstrap.cc:492] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Bootstrap starting.
I20260926 03:02:41.076829 14729 tablet_bootstrap.cc:492] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Bootstrap starting.
I20260926 03:02:41.079737 14729 tablet_bootstrap.cc:654] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:41.080152 14728 tablet_bootstrap.cc:654] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:41.080544 14730 tablet_bootstrap.cc:492] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Bootstrap starting.
I20260926 03:02:41.081036 14729 log.cc:824] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:41.081768 14728 log.cc:824] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:41.083875 14730 tablet_bootstrap.cc:654] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:41.084084 14729 tablet_bootstrap.cc:492] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: No bootstrap required, opened a new log
I20260926 03:02:41.084349 14729 ts_tablet_manager.cc:1397] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260926 03:02:41.084988 14730 log.cc:824] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:41.087827 14730 tablet_bootstrap.cc:492] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: No bootstrap required, opened a new log
I20260926 03:02:41.087862 14728 tablet_bootstrap.cc:492] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: No bootstrap required, opened a new log
I20260926 03:02:41.088074 14730 ts_tablet_manager.cc:1397] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 03:02:41.088074 14728 ts_tablet_manager.cc:1397] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260926 03:02:41.092571 14729 raft_consensus.cc:348] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.093092 14729 raft_consensus.cc:374] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:41.093219 14729 raft_consensus.cc:729] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1ab79108ff4d4bbb5ae3f3bf34e66e, State: Initialized, Role: FOLLOWER
I20260926 03:02:41.093967 14729 consensus_queue.cc:260] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.098448 14728 raft_consensus.cc:348] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.098905 14728 raft_consensus.cc:374] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:41.098948 14729 ts_tablet_manager.cc:1428] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 03:02:41.099013 14710 heartbeater.cc:503] Master 127.0.97.62:39161 was elected leader, sending a full tablet report...
I20260926 03:02:41.099026 14728 raft_consensus.cc:729] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0880de7f65149e3ac55a9d38583556f, State: Initialized, Role: FOLLOWER
I20260926 03:02:41.099617 14728 consensus_queue.cc:260] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.101588 14728 ts_tablet_manager.cc:1428] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260926 03:02:41.103809 14730 raft_consensus.cc:348] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.104277 14730 raft_consensus.cc:374] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:41.104383 14730 raft_consensus.cc:729] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb946f8ffcb452394eb6cff78ccf728, State: Initialized, Role: FOLLOWER
I20260926 03:02:41.104916 14730 consensus_queue.cc:260] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.106364 14577 heartbeater.cc:503] Master 127.0.97.62:39161 was elected leader, sending a full tablet report...
I20260926 03:02:41.106968 14730 ts_tablet_manager.cc:1428] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Time spent starting tablet: real 0.019s user 0.011s sys 0.003s
W20260926 03:02:41.210410 14711 tablet.cc:2369] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:02:41.232826 14578 tablet.cc:2369] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:02:41.233495 14444 tablet.cc:2369] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:02:41.317260 14734 raft_consensus.cc:482] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:41.317515 14734 raft_consensus.cc:504] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.319245 14734 leader_election.cc:288] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0880de7f65149e3ac55a9d38583556f (127.0.97.1:46139), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
I20260926 03:02:41.325033 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.319402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.97.3:52479)
0926 03:02:41.319703 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:41.319722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:41.319857 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:41.320501 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:41.320509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:41.320550 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:41.320843 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.320863 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.321884 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.321892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.323179 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.323190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.324268 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.324274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.324443 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.324466 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 03:02:41.324573 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:02:41.324740 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260926 03:02:41.325001 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.319631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52479 (local address 127.0.97.1:46139)
0926 03:02:41.319823 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.319829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.319848 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:02:41.320073 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.320082 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.320184 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.320286 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:41.321021 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.321752 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.323353 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.324107 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.324247 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.324344 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.324618 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.324745 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:02:41.326133 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.320222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33689 (local address 127.0.97.2:41303)
0926 03:02:41.320414 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.320420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.321104 (+ 684us) server_negotiation.cc:407] Connection header received
0926 03:02:41.321292 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.321295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.321354 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.321463 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:41.321948 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.322640 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.324267 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.325192 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.325316 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.325402 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.325764 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.325895 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:02:41.326176 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.319735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41303 (local address 127.0.97.3:33689)
0926 03:02:41.320959 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:41.320972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:41.321160 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:41.321475 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:41.321479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:41.321500 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:41.321799 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.321807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.322781 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.322786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.324120 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:41.324131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.325382 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.325389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:41.325586 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.325629 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:02:41.325737 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:02:41.325856 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"thread_start_us":926,"threads_started":1}
I20260926 03:02:41.326025 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60cfb17161841a3b1ab14142dd44903" candidate_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0880de7f65149e3ac55a9d38583556f" is_pre_election: true
I20260926 03:02:41.326601 14398 raft_consensus.cc:2454] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1ab79108ff4d4bbb5ae3f3bf34e66e in term 0.
I20260926 03:02:41.327302 14598 leader_election.cc:302] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f, fa1ab79108ff4d4bbb5ae3f3bf34e66e; no voters:
I20260926 03:02:41.327224 14532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60cfb17161841a3b1ab14142dd44903" candidate_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb946f8ffcb452394eb6cff78ccf728" is_pre_election: true
I20260926 03:02:41.327677 14734 raft_consensus.cc:2781] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:02:41.327786 14734 raft_consensus.cc:482] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:02:41.327771 14532 raft_consensus.cc:2454] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa1ab79108ff4d4bbb5ae3f3bf34e66e in term 0.
I20260926 03:02:41.327896 14734 raft_consensus.cc:3037] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:41.330669 14734 raft_consensus.cc:504] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.331311 14734 leader_election.cc:288] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [CANDIDATE]: Term 1 election: Requested vote from peers e0880de7f65149e3ac55a9d38583556f (127.0.97.1:46139), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
I20260926 03:02:41.331661 14398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60cfb17161841a3b1ab14142dd44903" candidate_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0880de7f65149e3ac55a9d38583556f"
I20260926 03:02:41.331763 14532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60cfb17161841a3b1ab14142dd44903" candidate_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb946f8ffcb452394eb6cff78ccf728"
I20260926 03:02:41.331856 14398 raft_consensus.cc:3037] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:41.331974 14532 raft_consensus.cc:3037] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:41.334200 14398 raft_consensus.cc:2454] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1ab79108ff4d4bbb5ae3f3bf34e66e in term 1.
I20260926 03:02:41.334331 14532 raft_consensus.cc:2454] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa1ab79108ff4d4bbb5ae3f3bf34e66e in term 1.
I20260926 03:02:41.334558 14598 leader_election.cc:302] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f, fa1ab79108ff4d4bbb5ae3f3bf34e66e; no voters:
I20260926 03:02:41.334861 14734 raft_consensus.cc:2781] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:02:41.335302 14734 raft_consensus.cc:686] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 LEADER]: Becoming Leader. State: Replica: fa1ab79108ff4d4bbb5ae3f3bf34e66e, State: Running, Role: LEADER
I20260926 03:02:41.335767 14734 consensus_queue.cc:237] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:41.339934 14253 catalog_manager.cc:5495] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } health_report { overall_health: UNKNOWN } } }
I20260926 03:02:41.404429 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.399037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.0.97.3:46881)
0926 03:02:41.399231 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.399238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.399258 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:02:41.399375 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.399378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.399462 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.399614 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:02:41.400013 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.400825 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.402314 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.402712 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.402849 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.403351 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:02:41.403870 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:02:41.404029 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.404086 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.404174 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:02:41.412145 14532 raft_consensus.cc:1261] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Refusing update from remote peer fa1ab79108ff4d4bbb5ae3f3bf34e66e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:41.412186 14398 raft_consensus.cc:1261] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Refusing update from remote peer fa1ab79108ff4d4bbb5ae3f3bf34e66e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:41.413087 14738 consensus_queue.cc:1035] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:41.413507 14734 consensus_queue.cc:1035] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:41.441498 14253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:02:41.441767 14253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:02:41.445251 14253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Sending DeleteTablet for 3 replicas of tablet a60cfb17161841a3b1ab14142dd44903
I20260926 03:02:41.446345 14378 tablet_service.cc:1514] Processing DeleteTablet for tablet a60cfb17161841a3b1ab14142dd44903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33532
I20260926 03:02:41.446671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14315
I20260926 03:02:41.446767 14645 tablet_service.cc:1514] Processing DeleteTablet for tablet a60cfb17161841a3b1ab14142dd44903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60406
I20260926 03:02:41.447438 14512 tablet_service.cc:1514] Processing DeleteTablet for tablet a60cfb17161841a3b1ab14142dd44903 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48258
I20260926 03:02:41.457373 14753 tablet_replica.cc:330] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: stopping tablet replica
I20260926 03:02:41.458549 14753 raft_consensus.cc:2229] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:02:41.459816 14753 raft_consensus.cc:2258] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:41.460820 14751 tablet_replica.cc:330] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: stopping tablet replica
I20260926 03:02:41.461746 14751 raft_consensus.cc:2229] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:02:41.462409 14751 raft_consensus.cc:2258] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:41.463665 14753 ts_tablet_manager.cc:1905] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:41.464638 14751 ts_tablet_manager.cc:1905] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:41.470875 14752 tablet_replica.cc:330] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: stopping tablet replica
I20260926 03:02:41.470911 14753 ts_tablet_manager.cc:1918] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:41.471046 14753 log.cc:1196] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/a60cfb17161841a3b1ab14142dd44903
I20260926 03:02:41.471256 14752 raft_consensus.cc:2229] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:02:41.471552 14753 ts_tablet_manager.cc:1939] T a60cfb17161841a3b1ab14142dd44903 P fbb946f8ffcb452394eb6cff78ccf728: Deleting consensus metadata
I20260926 03:02:41.471824 14752 raft_consensus.cc:2258] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:41.472259 14751 ts_tablet_manager.cc:1918] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:41.472465 14751 log.cc:1196] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/a60cfb17161841a3b1ab14142dd44903
I20260926 03:02:41.473044 14751 ts_tablet_manager.cc:1939] T a60cfb17161841a3b1ab14142dd44903 P e0880de7f65149e3ac55a9d38583556f: Deleting consensus metadata
I20260926 03:02:41.473342 14239 catalog_manager.cc:4841] TS fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303): tablet a60cfb17161841a3b1ab14142dd44903 (table test-table [id=dbe5cccf7326448494f9dcff37c38ece]) successfully deleted
I20260926 03:02:41.474790 14238 catalog_manager.cc:4841] TS e0880de7f65149e3ac55a9d38583556f (127.0.97.1:46139): tablet a60cfb17161841a3b1ab14142dd44903 (table test-table [id=dbe5cccf7326448494f9dcff37c38ece]) successfully deleted
Sep 26 03:02:41 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:02:41.476656 14752 ts_tablet_manager.cc:1905] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:41.482566 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.448868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.97.1:46139)
0926 03:02:41.449115 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.449123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.462776 (+ 13653us) server_negotiation.cc:407] Connection header received
0926 03:02:41.463223 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.463238 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.463339 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.463412 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:41.465666 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.466829 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.468793 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.469481 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.469603 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.476089 (+ 6486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:41.476100 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:41.476113 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:41.476147 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:41.479251 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.480085 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.480093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.480096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.480248 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.480717 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.480724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.480728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.481183 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:41.481462 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.482002 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.482213 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:02:41.484153 14752 ts_tablet_manager.cc:1918] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:41.484297 14438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:41.484326 14752 log.cc:1196] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/a60cfb17161841a3b1ab14142dd44903
I20260926 03:02:41.484922 14752 ts_tablet_manager.cc:1939] T a60cfb17161841a3b1ab14142dd44903 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Deleting consensus metadata
I20260926 03:02:41.487038 14239 catalog_manager.cc:4841] TS fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881): tablet a60cfb17161841a3b1ab14142dd44903 (table test-table [id=dbe5cccf7326448494f9dcff37c38ece]) successfully deleted
I20260926 03:02:41.582221 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14315
I20260926 03:02:41.599614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14449
Sep 26 03:02:41 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:02:41.615010 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.600453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.97.2:41303)
0926 03:02:41.600687 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.600694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.600714 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:02:41.600822 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.600827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.600916 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.601046 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:41.602325 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.603068 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.604352 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.604753 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.604881 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.609397 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:41.609410 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:41.609424 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:41.609462 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:41.612110 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.612763 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.612780 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.612782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.612920 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.613490 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.613495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.613504 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.613862 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:41.614160 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.614451 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.614671 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:02:41.615701 14572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:41.705054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14449
I20260926 03:02:41.728824 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14582
Sep 26 03:02:41 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:02:41.748256 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:41.729598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.0.97.3:46881)
0926 03:02:41.729776 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:41.729780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:41.729852 (+ 72us) server_negotiation.cc:407] Connection header received
0926 03:02:41.730042 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:41.730045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:41.730103 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:41.730193 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:41.731985 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.732975 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.735036 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:41.735485 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:41.735583 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:41.740987 (+ 5404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:41.741286 (+ 299us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:41.741310 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:41.741379 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:41.745359 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.746002 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.746011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.746015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.746225 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:41.746555 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:41.746566 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:41.746569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:41.747075 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:41.747424 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:41.747592 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:02:41.747820 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:02:41.749069 14705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:41.869165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14582
I20260926 03:02:41.888638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14222
I20260926 03:02:41.889770 14284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:41.998298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14222
I20260926 03:02:42.018879 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39161
--webserver_interface=127.0.97.62
--webserver_port=40689
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:42.333978 14759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:42.334223 14759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:42.334303 14759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:42.340920 14759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:42.341110 14759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:42.341173 14759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:42.341233 14759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:42.341275 14759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:42.350468 14759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:42.351653 14759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:42.352862 14759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:42.375079 14768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:42.375105 14765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:42.375175 14766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:42.375948 14759 server_base.cc:1034] running on GCE node
I20260926 03:02:42.376822 14759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:42.378448 14759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:42.379693 14759 hybrid_clock.cc:648] HybridClock initialized: now 1790391762379644 us; error 53 us; skew 500 ppm
I20260926 03:02:42.380088 14759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:42 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:42.383482 14759 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:02:42.385166 14759 webserver.cc:460] Webserver started at http://127.0.97.62:40689/ using document root <none> and password file <none>
I20260926 03:02:42.385973 14759 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:42.386137 14759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:42.391937 14759 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 03:02:42.394876 14775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:42.396185 14759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:02:42.396373 14759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "1348d1e584534fb0bc12358a2d807247"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "12c587fff2e63b120b7e5a3b63db080c"
server_key_iv: "dca6c744a1dd5ace7098b4163c7e66e3"
server_key_version: "encryptionkey@0"
I20260926 03:02:42.397456 14759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:42.413933 14759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:42.417748 14759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:42.418031 14759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:42.436677 14759 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39161
I20260926 03:02:42.436703 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39161 every 8 connection(s)
I20260926 03:02:42.438158 14759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:02:42.440961 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14759
I20260926 03:02:42.441977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46139
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39161
--webserver_port=42581
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:02:42.447093 14827 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:02:42.451587 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Bootstrap starting.
I20260926 03:02:42.471014 14827 log.cc:824] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:42.482983 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:02:42.483738 14827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Bootstrap complete.
I20260926 03:02:42.505770 14827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:42.507368 14827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1348d1e584534fb0bc12358a2d807247, State: Initialized, Role: FOLLOWER
I20260926 03:02:42.508167 14827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:42.508489 14827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:02:42.508632 14827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:02:42.508853 14827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:02:42.515941 14827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:42.516592 14827 leader_election.cc:302] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1348d1e584534fb0bc12358a2d807247; no voters:
I20260926 03:02:42.517407 14827 leader_election.cc:288] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:02:42.517573 14831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:02:42.524766 14831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 LEADER]: Becoming Leader. State: Replica: 1348d1e584534fb0bc12358a2d807247, State: Running, Role: LEADER
I20260926 03:02:42.525522 14831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:42.526391 14827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:42.533058 14833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1348d1e584534fb0bc12358a2d807247. Latest consensus state: current_term: 2 leader_uuid: "1348d1e584534fb0bc12358a2d807247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } } }
I20260926 03:02:42.533541 14833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:42.535698 14832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1348d1e584534fb0bc12358a2d807247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } } }
I20260926 03:02:42.536787 14832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:42.541922 14839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:02:42.553622 14839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:02:42.554986 14839 catalog_manager.cc:1259] Loaded cluster ID: 3834cc9eb8644a4c8b3f52f4b4249c1a
I20260926 03:02:42.555080 14839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:02:42.558493 14839 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 03:02:42.577970 14839 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 03:02:42.587380 14839 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:02:42.589112 14839 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Loaded TSK: 0
I20260926 03:02:42.590895 14839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:02:42.606439 14850 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260926 03:02:42.900792 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:42.901122 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:42.901290 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:42.908517 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:42.908661 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:42.908878 14829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:02:42.918339 14829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42581
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:42.919595 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:42.920908 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:42.934377 14855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:42.934404 14856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:42.935027 14858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:42.935446 14829 server_base.cc:1034] running on GCE node
I20260926 03:02:42.936102 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:42.937194 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:42.938416 14829 hybrid_clock.cc:648] HybridClock initialized: now 1790391762938354 us; error 52 us; skew 500 ppm
I20260926 03:02:42.938772 14829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:42 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:42.942101 14829 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:02:42.943486 14829 webserver.cc:460] Webserver started at http://127.0.97.1:42581/ using document root <none> and password file <none>
I20260926 03:02:42.944150 14829 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:42.944298 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:42.949361 14829 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:02:42.951813 14865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:42.953107 14829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:02:42.953272 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "e0880de7f65149e3ac55a9d38583556f"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "a0078a41db700e4eabb1e425c313b3d2"
server_key_iv: "cd54f23c823250029223f1ca87b7faa2"
server_key_version: "encryptionkey@0"
I20260926 03:02:42.954176 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:42.968185 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:42.971922 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:42.972172 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:42.973053 14829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:42.974237 14829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:42.974336 14829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:42.974479 14829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:42.974578 14829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:43.005486 14829 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46139
I20260926 03:02:43.005568 14977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46139 every 8 connection(s)
I20260926 03:02:43.006932 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:02:43.016986 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14829
I20260926 03:02:43.017614 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41303
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39161
--webserver_port=37077
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:02:43 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:43.029582 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.009633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43201 (local address 127.0.97.62:39161)
0926 03:02:43.010284 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.010300 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.010662 (+ 362us) server_negotiation.cc:407] Connection header received
0926 03:02:43.012677 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.012689 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.012991 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.013399 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:43.014775 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.015954 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.017525 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.018194 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.018421 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.021803 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:43.021811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:43.021824 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:43.021859 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:43.025303 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.026038 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.026047 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.026061 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.026207 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.026697 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.026703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.026706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.027209 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:43.027426 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.028177 (+ 751us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.028571 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260926 03:02:43.029960 14981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.009749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:43201)
0926 03:02:43.010506 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:43.010542 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:43.012177 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:43.013562 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:43.013569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:43.014078 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:43.014576 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.014592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.016095 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.016099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.017377 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.017389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.018439 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.018446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.018705 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.019385 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:43.019406 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:43.021542 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:43.025455 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:43.025464 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:43.025478 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:43.025904 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:43.026327 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:43.026334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:43.026338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:43.026588 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:43.027350 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:43.027358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:43.027712 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 03:02:43.027975 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 03:02:43.028548 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":305,"threads_started":1}
I20260926 03:02:43.032759 14978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39161
I20260926 03:02:43.033104 14978 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:43.033978 14978 heartbeater.cc:511] Master 127.0.97.62:39161 requested a full tablet report, sending...
I20260926 03:02:43.036368 14792 ts_manager.cc:194] Registered new tserver with Master: e0880de7f65149e3ac55a9d38583556f (127.0.97.1:46139)
I20260926 03:02:43.037976 14792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43201
W20260926 03:02:43.338116 14982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:43.338356 14982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:43.338471 14982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:43.345285 14982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:43.345396 14982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:43.345577 14982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:02:43.354760 14982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37077
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:43.355942 14982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:43.357275 14982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:43.370034 14990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:43.370087 14988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:43.369988 14987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:43.371019 14982 server_base.cc:1034] running on GCE node
I20260926 03:02:43.371544 14982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:43.372378 14982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:43.373606 14982 hybrid_clock.cc:648] HybridClock initialized: now 1790391763373561 us; error 54 us; skew 500 ppm
I20260926 03:02:43.373936 14982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:43 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:43.377058 14982 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:02:43.378419 14982 webserver.cc:460] Webserver started at http://127.0.97.2:37077/ using document root <none> and password file <none>
I20260926 03:02:43.379052 14982 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:43.379236 14982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:43.384194 14982 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:02:43.386730 14997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:43.387946 14982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:02:43.388111 14982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "fbb946f8ffcb452394eb6cff78ccf728"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "6feb70b1e845d2109a39f34556abeadc"
server_key_iv: "a888605fe753471c7732b79385868fd3"
server_key_version: "encryptionkey@0"
I20260926 03:02:43.388995 14982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:43.405663 14982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:43.409727 14982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:43.409999 14982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:43.410902 14982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:43.412149 14982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:43.412276 14982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:43.412415 14982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:43.412509 14982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:43.445186 14982 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41303
I20260926 03:02:43.445241 15109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41303 every 8 connection(s)
I20260926 03:02:43.446630 14982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:02:43.451383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 14982
I20260926 03:02:43.457167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46881
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39161
--webserver_port=37141
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:02:43 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:43.467443 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.449246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54081 (local address 127.0.97.62:39161)
0926 03:02:43.449429 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.449433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.450622 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 03:02:43.452597 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.452603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.452726 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.452836 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:43.454267 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.455416 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.456878 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.457473 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.457606 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.461094 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:43.461103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:43.461111 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:43.461155 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:43.464302 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.465197 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.465201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.465203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.465311 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.465790 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.465793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.465795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.466050 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:43.466156 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.466946 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.467144 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:02:43.468948 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.449761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:54081)
0926 03:02:43.450475 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:43.450510 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:43.452330 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:43.453093 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:43.453101 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:43.453568 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:43.454095 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.454110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.455563 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.455568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.456712 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.456722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.457693 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.457698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.457944 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.458578 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:43.458597 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:43.460834 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:43.464450 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:43.464459 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:43.464477 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:43.465091 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:43.465424 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:43.465432 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:43.465438 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:43.465676 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:43.466168 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:43.466178 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:43.466592 (+ 414us) client_negotiation.cc:769] Sending connection context
0926 03:02:43.466876 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 03:02:43.467535 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":297,"threads_started":1}
I20260926 03:02:43.470446 15110 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39161
I20260926 03:02:43.470743 15110 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:43.471609 15110 heartbeater.cc:511] Master 127.0.97.62:39161 requested a full tablet report, sending...
I20260926 03:02:43.473137 14792 ts_manager.cc:194] Registered new tserver with Master: fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
I20260926 03:02:43.473912 14792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54081
W20260926 03:02:43.773434 15113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:43.773731 15113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:43.773922 15113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:43.780857 15113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:43.780995 15113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:43.781206 15113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:02:43.790488 15113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37141
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:43.791786 15113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:43.793279 15113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:43.805825 15121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:43.805920 15118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:43.805895 15119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:43.806814 15113 server_base.cc:1034] running on GCE node
I20260926 03:02:43.807413 15113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:43.808291 15113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:43.809521 15113 hybrid_clock.cc:648] HybridClock initialized: now 1790391763809480 us; error 64 us; skew 500 ppm
I20260926 03:02:43.809826 15113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:43 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:43.812757 15113 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:02:43.814153 15113 webserver.cc:460] Webserver started at http://127.0.97.3:37141/ using document root <none> and password file <none>
I20260926 03:02:43.814844 15113 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:43.815001 15113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:43.820205 15113 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:02:43.822839 15128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:43.824071 15113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:02:43.824241 15113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "9b912c59fa722a6e729c57e9c6028a74"
server_key_iv: "76d5cbee2b1f5964b74aedafc13bf3f4"
server_key_version: "encryptionkey@0"
I20260926 03:02:43.825265 15113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:43.858807 15113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:43.862749 15113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:43.863013 15113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:43.863955 15113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:43.865231 15113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:43.865378 15113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:43.865502 15113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:43.865634 15113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:43.897993 15113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46881
I20260926 03:02:43.898078 15240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46881 every 8 connection(s)
I20260926 03:02:43.899441 15113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:02:43.901527 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 15113
Sep 26 03:02:43 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:02:43.918202 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.902457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.0.97.62:39161)
0926 03:02:43.902915 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.902923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.905583 (+ 2660us) server_negotiation.cc:407] Connection header received
0926 03:02:43.905788 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.905792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.905873 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.905977 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:43.907402 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.908141 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.909511 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.909993 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.910141 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.912830 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:43.912842 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:43.912851 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:43.912880 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:43.915490 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.916148 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.916153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.916156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.916273 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.916636 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.916641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.916645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.916948 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:43.917269 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.917585 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.917791 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"spinlock_wait_cycles":1920,"thread_start_us":245,"threads_started":1}
I20260926 03:02:43.918875 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.901908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51457 (local address 127.0.97.62:39161)
0926 03:02:43.902120 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.902129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.903435 (+ 1306us) server_negotiation.cc:407] Connection header received
0926 03:02:43.905309 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.905314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.905402 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.905511 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:43.906907 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.907639 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.909237 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.909661 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.909835 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.913100 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:43.913106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:43.913110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:43.913158 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:43.915880 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.916557 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.916562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.916565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.916693 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:43.917162 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:43.917167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:43.917170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:43.917433 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:43.917577 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.918276 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.918505 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"spinlock_wait_cycles":2944}
I20260926 03:02:43.920326 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.902504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:51457)
0926 03:02:43.903289 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:43.903330 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:43.904975 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:43.905749 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:43.905758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:43.906223 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:43.906745 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.906761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.907833 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.907839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.909103 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.909113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.909919 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.909927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.910181 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.910840 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:43.910870 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:43.912905 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:43.916020 (+ 3115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:43.916028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:43.916042 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:43.916437 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:43.916832 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:43.916837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:43.916841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:43.917062 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:43.917554 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:43.917562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:43.917865 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 03:02:43.918135 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 03:02:43.918643 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":365,"threads_started":1}
I20260926 03:02:43.921577 15241 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39161
I20260926 03:02:43.921830 15241 heartbeater.cc:464] Registering TS with master...
I20260926 03:02:43.922452 15241 heartbeater.cc:511] Master 127.0.97.62:39161 requested a full tablet report, sending...
I20260926 03:02:43.923770 14792 ts_manager.cc:194] Registered new tserver with Master: fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881)
I20260926 03:02:43.924546 14792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51457
I20260926 03:02:43.932658 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:02:43.938901 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.933582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.0.97.62:39161)
0926 03:02:43.933733 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.933737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.934200 (+ 463us) server_negotiation.cc:407] Connection header received
0926 03:02:43.934366 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.934370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.934492 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.934616 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:02:43.934986 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.935643 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.937076 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.937442 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.937528 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.937983 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:02:43.938407 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:02:43.938508 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.938651 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.938707 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:02:43.939702 14792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:02:43.946652 14792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:02:43.976286 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.957337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48278)
0926 03:02:43.967043 (+ 9706us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:43.967078 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:43.967247 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:43.968100 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:43.968109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:43.968146 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:43.968719 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.968738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.971070 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.971078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.973447 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.973461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.975331 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.975337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.975538 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.975592 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 03:02:43.975721 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 03:02:43.975904 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9469,"thread_start_us":9306,"threads_started":1}
I20260926 03:02:43.976290 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.957489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.97.1:46139)
0926 03:02:43.966565 (+ 9076us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.966576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.967434 (+ 858us) server_negotiation.cc:407] Connection header received
0926 03:02:43.967612 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.967620 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.967878 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.968737 (+ 859us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:43.968969 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.970952 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.973655 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.975129 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.975265 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.975455 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.975788 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.975940 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8904,"thread_start_us":8795,"threads_started":1}
I20260926 03:02:43.977598 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.957598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46881 (local address 127.0.0.1:52360)
0926 03:02:43.969034 (+ 11436us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:43.969054 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:43.969164 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:43.970594 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:43.970602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:43.970638 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:43.971084 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.971098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.972990 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.972996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.974920 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.974931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.976713 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.976719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.976892 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.976924 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:02:43.977101 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 03:02:43.977219 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11293,"mutex_wait_us":2,"thread_start_us":11137,"threads_started":1}
I20260926 03:02:43.977921 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.958216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.0.97.3:46881)
0926 03:02:43.968629 (+ 10413us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.968639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.969621 (+ 982us) server_negotiation.cc:407] Connection header received
0926 03:02:43.969740 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.969750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.970061 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.970272 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:43.971477 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.972847 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.975076 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.976498 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.976623 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.976840 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.977144 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.977291 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10235,"thread_start_us":10048,"threads_started":1}
I20260926 03:02:43.978976 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.969190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41303 (local address 127.0.0.1:42040)
0926 03:02:43.969729 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:43.969748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:43.969856 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:43.971971 (+ 2115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:43.971976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:43.971999 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:43.972418 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.972430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.974249 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.974254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.976204 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:43.976218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.978234 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.978241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:43.978427 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.978457 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:02:43.978549 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:02:43.978662 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":246,"threads_started":1}
I20260926 03:02:43.979560 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:43.969476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.0.97.2:41303)
0926 03:02:43.971459 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:43.971467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:43.971496 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:02:43.971578 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:43.971584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:43.971775 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:43.971959 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:43.972604 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.974087 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.976631 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:43.978026 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:43.978150 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:43.978331 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:43.979122 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 03:02:43.979296 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1831,"thread_start_us":1718,"threads_started":1}
I20260926 03:02:43.980510 14913 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5a3a5dd1994452b921c2620804b6b7 (DEFAULT_TABLE table=test-table [id=aef4763ec1634acba66998168b9e8eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:43.981662 14913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5a3a5dd1994452b921c2620804b6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:43.981396 15176 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5a3a5dd1994452b921c2620804b6b7 (DEFAULT_TABLE table=test-table [id=aef4763ec1634acba66998168b9e8eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:43.982749 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5a3a5dd1994452b921c2620804b6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:43.982833 15045 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5a3a5dd1994452b921c2620804b6b7 (DEFAULT_TABLE table=test-table [id=aef4763ec1634acba66998168b9e8eed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:02:43.984042 15045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5a3a5dd1994452b921c2620804b6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:02:43.993880 15257 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Bootstrap starting.
I20260926 03:02:43.997165 15257 tablet_bootstrap.cc:654] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:43.997574 15258 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Bootstrap starting.
I20260926 03:02:43.998519 15257 log.cc:824] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:43.998980 15259 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Bootstrap starting.
I20260926 03:02:44.000841 15258 tablet_bootstrap.cc:654] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:44.002139 15259 tablet_bootstrap.cc:654] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Neither blocks nor log segments found. Creating new log.
I20260926 03:02:44.002128 15258 log.cc:824] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:44.002687 15257 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: No bootstrap required, opened a new log
I20260926 03:02:44.002977 15257 ts_tablet_manager.cc:1397] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 03:02:44.003309 15259 log.cc:824] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:44.005937 15258 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: No bootstrap required, opened a new log
I20260926 03:02:44.006057 15259 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: No bootstrap required, opened a new log
I20260926 03:02:44.006238 15258 ts_tablet_manager.cc:1397] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 03:02:44.006337 15259 ts_tablet_manager.cc:1397] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260926 03:02:44.014660 15257 raft_consensus.cc:348] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.015189 15257 raft_consensus.cc:374] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:44.015362 15257 raft_consensus.cc:729] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0880de7f65149e3ac55a9d38583556f, State: Initialized, Role: FOLLOWER
I20260926 03:02:44.015861 15259 raft_consensus.cc:348] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.016305 15259 raft_consensus.cc:374] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:44.016160 15257 consensus_queue.cc:260] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.016438 15259 raft_consensus.cc:729] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb946f8ffcb452394eb6cff78ccf728, State: Initialized, Role: FOLLOWER
I20260926 03:02:44.017095 15259 consensus_queue.cc:260] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.017542 15258 raft_consensus.cc:348] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.018036 15258 raft_consensus.cc:374] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:02:44.018213 15258 raft_consensus.cc:729] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa1ab79108ff4d4bbb5ae3f3bf34e66e, State: Initialized, Role: FOLLOWER
I20260926 03:02:44.019006 15258 consensus_queue.cc:260] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.024628 15257 ts_tablet_manager.cc:1428] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Time spent starting tablet: real 0.022s user 0.009s sys 0.009s
I20260926 03:02:44.025148 14978 heartbeater.cc:503] Master 127.0.97.62:39161 was elected leader, sending a full tablet report...
I20260926 03:02:44.027587 15110 heartbeater.cc:503] Master 127.0.97.62:39161 was elected leader, sending a full tablet report...
I20260926 03:02:44.028010 15241 heartbeater.cc:503] Master 127.0.97.62:39161 was elected leader, sending a full tablet report...
I20260926 03:02:44.028138 15259 ts_tablet_manager.cc:1428] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Time spent starting tablet: real 0.022s user 0.017s sys 0.003s
I20260926 03:02:44.030975 15258 ts_tablet_manager.cc:1428] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Time spent starting tablet: real 0.025s user 0.012s sys 0.008s
I20260926 03:02:44.041258 15263 raft_consensus.cc:482] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:44.041482 15263 raft_consensus.cc:504] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.042912 15263 leader_election.cc:288] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
I20260926 03:02:44.050021 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.043219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45063 (local address 127.0.97.3:46881)
0926 03:02:44.043447 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:44.043456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:44.044030 (+ 574us) server_negotiation.cc:407] Connection header received
0926 03:02:44.044262 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:44.044293 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:44.044366 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:44.044448 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:44.045216 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.046133 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.048250 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.049055 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.049202 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.049306 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:44.049632 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 03:02:44.049757 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:02:44.050091 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.043269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46881 (local address 127.0.97.1:45063)
0926 03:02:44.043892 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.043932 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:44.044079 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:44.044628 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:44.044631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:44.044651 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:44.045071 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:44.045084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.046259 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.046272 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:44.048087 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:44.048098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.049252 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.049258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:44.049424 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.049448 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:02:44.049606 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 03:02:44.049731 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":347,"threads_started":1}
I20260926 03:02:44.050138 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.044218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49085 (local address 127.0.97.2:41303)
0926 03:02:44.044423 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:44.044431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:44.044721 (+ 290us) server_negotiation.cc:407] Connection header received
0926 03:02:44.044859 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:44.044866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:44.044935 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:44.045068 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:02:44.045669 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.046471 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.048404 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.049216 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.049365 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.049462 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:44.049741 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 03:02:44.049872 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:02:44.050218 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.043969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41303 (local address 127.0.97.1:49085)
0926 03:02:44.044620 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.044634 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:44.044739 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:44.045089 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:44.045093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:44.045111 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:02:44.045516 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:44.045529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.046609 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.046614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:44.048241 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:44.048253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.049445 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.049451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:44.049591 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.049613 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:02:44.049716 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:02:44.049838 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":340,"threads_started":1}
I20260926 03:02:44.051172 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5a3a5dd1994452b921c2620804b6b7" candidate_uuid: "e0880de7f65149e3ac55a9d38583556f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb946f8ffcb452394eb6cff78ccf728" is_pre_election: true
I20260926 03:02:44.051198 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5a3a5dd1994452b921c2620804b6b7" candidate_uuid: "e0880de7f65149e3ac55a9d38583556f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" is_pre_election: true
I20260926 03:02:44.051616 15065 raft_consensus.cc:2454] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0880de7f65149e3ac55a9d38583556f in term 0.
I20260926 03:02:44.051615 15196 raft_consensus.cc:2454] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0880de7f65149e3ac55a9d38583556f in term 0.
I20260926 03:02:44.052218 14867 leader_election.cc:302] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f, fa1ab79108ff4d4bbb5ae3f3bf34e66e; no voters:
I20260926 03:02:44.052597 15263 raft_consensus.cc:2781] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:02:44.052731 15263 raft_consensus.cc:482] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:02:44.052860 15263 raft_consensus.cc:3037] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:44.054997 15263 raft_consensus.cc:504] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.055514 15263 leader_election.cc:288] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 1 election: Requested vote from peers fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
I20260926 03:02:44.055919 15196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5a3a5dd1994452b921c2620804b6b7" candidate_uuid: "e0880de7f65149e3ac55a9d38583556f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e"
I20260926 03:02:44.055962 15065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5a3a5dd1994452b921c2620804b6b7" candidate_uuid: "e0880de7f65149e3ac55a9d38583556f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb946f8ffcb452394eb6cff78ccf728"
I20260926 03:02:44.056087 15196 raft_consensus.cc:3037] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:44.056180 15065 raft_consensus.cc:3037] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:02:44.058462 15065 raft_consensus.cc:2454] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0880de7f65149e3ac55a9d38583556f in term 1.
I20260926 03:02:44.058586 15196 raft_consensus.cc:2454] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0880de7f65149e3ac55a9d38583556f in term 1.
I20260926 03:02:44.058818 14867 leader_election.cc:302] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f, fbb946f8ffcb452394eb6cff78ccf728; no voters:
I20260926 03:02:44.059073 15263 raft_consensus.cc:2781] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:02:44.059500 15263 raft_consensus.cc:686] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 LEADER]: Becoming Leader. State: Replica: e0880de7f65149e3ac55a9d38583556f, State: Running, Role: LEADER
I20260926 03:02:44.059928 15263 consensus_queue.cc:237] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:44.064823 14790 catalog_manager.cc:5495] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f reported cstate change: term changed from 0 to 1, leader changed from <none> to e0880de7f65149e3ac55a9d38583556f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e0880de7f65149e3ac55a9d38583556f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } health_report { overall_health: UNKNOWN } } }
I20260926 03:02:44.102932 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.097517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.0.97.1:46139)
0926 03:02:44.097710 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:44.097716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:44.097748 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:02:44.097848 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:44.097852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:44.097924 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:44.098069 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:02:44.098434 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.099173 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.100688 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.101227 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.101361 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.101851 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:02:44.102380 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:02:44.102530 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:44.102598 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 03:02:44.102670 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:02:44.110776 15196 raft_consensus.cc:1261] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 FOLLOWER]: Refusing update from remote peer e0880de7f65149e3ac55a9d38583556f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:44.110910 15065 raft_consensus.cc:1261] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Refusing update from remote peer e0880de7f65149e3ac55a9d38583556f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:02:44.111779 15268 consensus_queue.cc:1035] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:44.112130 15263 consensus_queue.cc:1035] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:02:44.122972 15271 mvcc.cc:205] Tried to move back new op lower bound from 7333444665787883520 to 7333444665593221120. Current Snapshot: MvccSnapshot[applied={T|T < 7333444665787883520}]
I20260926 03:02:44.136113 14790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:02:44.136421 14790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:02:44.140954 14790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Sending DeleteTablet for 3 replicas of tablet 6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:44.142580 15176 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5a3a5dd1994452b921c2620804b6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52360
I20260926 03:02:44.142604 14913 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5a3a5dd1994452b921c2620804b6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48278
I20260926 03:02:44.143494 15045 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5a3a5dd1994452b921c2620804b6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:02:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42040
I20260926 03:02:44.143874 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14829
I20260926 03:02:44.144207 15281 tablet_replica.cc:330] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: stopping tablet replica
I20260926 03:02:44.144671 15281 raft_consensus.cc:2229] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:02:44.145298 15281 raft_consensus.cc:2258] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:44.146207 15283 tablet_replica.cc:330] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: stopping tablet replica
I20260926 03:02:44.146646 15283 raft_consensus.cc:2229] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:02:44.147156 15283 raft_consensus.cc:2258] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:02:44.148695 15281 ts_tablet_manager.cc:1905] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:44.152515 15283 ts_tablet_manager.cc:1905] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:02:44.155848 15281 ts_tablet_manager.cc:1918] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:44.156041 15281 log.cc:1196] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:44.156647 15281 ts_tablet_manager.cc:1939] T 6d5a3a5dd1994452b921c2620804b6b7 P fa1ab79108ff4d4bbb5ae3f3bf34e66e: Deleting consensus metadata
I20260926 03:02:44.158588 14777 catalog_manager.cc:4841] TS fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881): tablet 6d5a3a5dd1994452b921c2620804b6b7 (table test-table [id=aef4763ec1634acba66998168b9e8eed]) successfully deleted
I20260926 03:02:44.163676 15252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.145355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.97.1:46139)
0926 03:02:44.145559 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:44.145566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:44.146593 (+ 1027us) server_negotiation.cc:407] Connection header received
0926 03:02:44.146688 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:44.146693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:44.146851 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:44.147009 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:44.148790 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.150072 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.151861 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.152550 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.152669 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.155917 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:44.155930 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:44.155951 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:44.155992 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:44.160161 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:44.161159 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:44.161167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:44.161171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:44.161357 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:44.161857 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:44.161863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:44.161866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:44.162267 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:44.162585 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:44.163033 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 03:02:44.163257 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:02:44.164538 14973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:44.168339 15283 ts_tablet_manager.cc:1918] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:02:44.168506 15283 log.cc:1196] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:44.169163 15283 ts_tablet_manager.cc:1939] T 6d5a3a5dd1994452b921c2620804b6b7 P fbb946f8ffcb452394eb6cff78ccf728: Deleting consensus metadata
I20260926 03:02:44.170861 14777 catalog_manager.cc:4841] TS fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303): tablet 6d5a3a5dd1994452b921c2620804b6b7 (table test-table [id=aef4763ec1634acba66998168b9e8eed]) successfully deleted
I20260926 03:02:44.259285 15282 tablet_replica.cc:330] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: stopping tablet replica
I20260926 03:02:44.259496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14829
I20260926 03:02:44.277621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14982
W20260926 03:02:44.277607 14776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46139 (error 108)
I20260926 03:02:44.279419 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.278949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48308)
0926 03:02:44.279144 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.279268 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:02:44.279619 14776 catalog_manager.cc:4584] TS e0880de7f65149e3ac55a9d38583556f (127.0.97.1:46139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d5a3a5dd1994452b921c2620804b6b7: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
I20260926 03:02:44.290990 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.278415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.0.97.2:41303)
0926 03:02:44.278597 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:44.278603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:44.278732 (+ 129us) server_negotiation.cc:407] Connection header received
0926 03:02:44.278945 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:44.278951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:44.279056 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:44.279158 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:44.280535 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.281340 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.282621 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.283000 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.283086 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.285606 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:44.285619 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:44.285633 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:44.285676 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:44.288524 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:44.289225 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:44.289231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:44.289233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:44.289354 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:44.289665 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:44.289669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:44.289671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:44.290017 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:44.290268 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:44.290482 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 03:02:44.290641 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:02:44.291742 15105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:44.333278 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.332762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48312)
0926 03:02:44.332983 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.333135 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:02:44.334242 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.333929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48320)
0926 03:02:44.334064 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.334129 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:02:44.378872 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14982
I20260926 03:02:44.394851 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.394199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48322)
0926 03:02:44.394488 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.394604 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 03:02:44.395920 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.395523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48332)
0926 03:02:44.395693 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.395779 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:02:44.396399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 15113
I20260926 03:02:44.415695 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.397041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.0.97.3:46881)
0926 03:02:44.397212 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:44.397216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:44.397249 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:02:44.397375 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:44.397385 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:44.397479 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:44.397558 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:44.398850 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.399601 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.400940 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:44.401580 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:44.401671 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:44.410741 (+ 9070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:44.410755 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:44.410772 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:44.410806 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:44.413418 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:44.414079 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:44.414086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:44.414088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:44.414215 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:44.414500 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:44.414503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:44.414505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:44.414859 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:44.415058 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:44.415287 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:02:44.415453 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:02:44.416298 15236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:44.476724 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.476137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48346)
0926 03:02:44.476397 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.476506 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 03:02:44.477870 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:44.477475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46139 (local address 127.0.0.1:48360)
0926 03:02:44.477698 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:44.477765 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 03:02:44.504293 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 15113
I20260926 03:02:44.520372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 14759
I20260926 03:02:44.521463 14822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:02:44.634603 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 14759
I20260926 03:02:44.656030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39161
--webserver_interface=127.0.97.62
--webserver_port=40689
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:44.974359 15289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:44.974637 15289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:44.974764 15289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:44.981660 15289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:02:44.981762 15289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:44.981863 15289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:44.981995 15289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:02:44.982091 15289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:02:44.992170 15289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:44.993494 15289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:44.994704 15289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:45.005815 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:45.005942 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:45.005702 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:45.006914 15289 server_base.cc:1034] running on GCE node
I20260926 03:02:45.007565 15289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:45.008802 15289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:45.010010 15289 hybrid_clock.cc:648] HybridClock initialized: now 1790391765009960 us; error 48 us; skew 500 ppm
I20260926 03:02:45.010351 15289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:45 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:45.013397 15289 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:02:45.014869 15289 webserver.cc:460] Webserver started at http://127.0.97.62:40689/ using document root <none> and password file <none>
I20260926 03:02:45.015467 15289 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:45.015568 15289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:45.020494 15289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:02:45.023123 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:45.024372 15289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:02:45.024544 15289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "1348d1e584534fb0bc12358a2d807247"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "12c587fff2e63b120b7e5a3b63db080c"
server_key_iv: "dca6c744a1dd5ace7098b4163c7e66e3"
server_key_version: "encryptionkey@0"
I20260926 03:02:45.025548 15289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:45.039574 15289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:45.043259 15289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:45.043526 15289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:45.061412 15289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39161
I20260926 03:02:45.061455 15355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39161 every 8 connection(s)
I20260926 03:02:45.062892 15289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:02:45.067243 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 15289
I20260926 03:02:45.068228 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46139
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39161
--webserver_port=42581
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:02:45.070531 15356 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:02:45.075408 15356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Bootstrap starting.
I20260926 03:02:45.098275 15356 log.cc:824] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:45.113785 15356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:02:45.114301 15356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247: Bootstrap complete.
I20260926 03:02:45.123027 15356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:45.123924 15356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1348d1e584534fb0bc12358a2d807247, State: Initialized, Role: FOLLOWER
I20260926 03:02:45.124547 15356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:45.124791 15356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:02:45.124914 15356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:02:45.125113 15356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:02:45.128726 15356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:45.129232 15356 leader_election.cc:302] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1348d1e584534fb0bc12358a2d807247; no voters:
I20260926 03:02:45.129885 15356 leader_election.cc:288] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:02:45.130017 15360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:02:45.132118 15356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:02:45.137039 15360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [term 3 LEADER]: Becoming Leader. State: Replica: 1348d1e584534fb0bc12358a2d807247, State: Running, Role: LEADER
I20260926 03:02:45.137720 15360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } }
I20260926 03:02:45.140475 15361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1348d1e584534fb0bc12358a2d807247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } } }
I20260926 03:02:45.140844 15361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:45.141563 15362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1348d1e584534fb0bc12358a2d807247. Latest consensus state: current_term: 3 leader_uuid: "1348d1e584534fb0bc12358a2d807247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1348d1e584534fb0bc12358a2d807247" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39161 } } }
I20260926 03:02:45.141897 15362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1348d1e584534fb0bc12358a2d807247 [sys.catalog]: This master's current role is: LEADER
I20260926 03:02:45.144100 15372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:02:45.151577 15372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:02:45.153409 15372 catalog_manager.cc:1259] Loaded cluster ID: 3834cc9eb8644a4c8b3f52f4b4249c1a
I20260926 03:02:45.153600 15372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 03:02:45.156102 15372 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff68fa049ad google::LogMessage::Fail() at ??:0
@ 0x7ff68fa0788a google::LogMessage::SendToLog() at ??:0
@ 0x7ff68fa044de google::LogMessage::Flush() at ??:0
@ 0x7ff68fa05a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff6a5437512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff6a5434976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff6a54bc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff6a54bc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff692650345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff69265391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff692653799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff69262eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff6959086db start_thread at ??:0
@ 0x7ff68e89e61f clone at ??:0
W20260926 03:02:45.444905 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:45.445241 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:45.445338 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:45.452442 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:45.452533 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:45.452638 15358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:02:45.461735 15358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42581
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:45.462831 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:45.463989 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:45.476886 15398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:45.476975 15401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:45.477085 15399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:45.478401 15358 server_base.cc:1034] running on GCE node
I20260926 03:02:45.479039 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:45.479945 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:45.481086 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790391765481036 us; error 57 us; skew 500 ppm
I20260926 03:02:45.481494 15358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:45 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:45.485476 15358 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:02:45.490247 15358 webserver.cc:460] Webserver started at http://127.0.97.1:42581/ using document root <none> and password file <none>
I20260926 03:02:45.491142 15358 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:45.491302 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:45.498224 15358 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 03:02:45.502010 15408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:45.503927 15358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:02:45.504096 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "e0880de7f65149e3ac55a9d38583556f"
format_stamp: "Formatted at 2026-09-26 03:02:39 on dist-test-slave-fccw"
server_key: "a0078a41db700e4eabb1e425c313b3d2"
server_key_iv: "cd54f23c823250029223f1ca87b7faa2"
server_key_version: "encryptionkey@0"
I20260926 03:02:45.505403 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:45.531791 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:45.536049 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:45.536324 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:45.537293 15358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:45.538955 15415 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 03:02:45.544469 15358 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 03:02:45.544603 15358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 03:02:45.544749 15358 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 03:02:45.547782 15358 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 03:02:45.547876 15358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260926 03:02:45.548197 15415 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Bootstrap starting.
I20260926 03:02:45.574981 15415 log.cc:824] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:02:45.588460 15415 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:02:45.589090 15415 tablet_bootstrap.cc:492] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Bootstrap complete.
I20260926 03:02:45.590750 15415 ts_tablet_manager.cc:1397] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Time spent bootstrapping tablet: real 0.043s user 0.027s sys 0.013s
I20260926 03:02:45.592700 15358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46139
I20260926 03:02:45.592734 15522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46139 every 8 connection(s)
I20260926 03:02:45.594129 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:02:45.598904 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:45.597031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:46329)
0926 03:02:45.597839 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:45.598052 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":363,"threads_started":1}
W20260926 03:02:45.599561 15523 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39161 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39161: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
I20260926 03:02:45.600387 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:45.600075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:48359)
0926 03:02:45.600210 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:45.600273 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:02:45.599392 15415 raft_consensus.cc:348] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:45.600596 15415 raft_consensus.cc:729] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0880de7f65149e3ac55a9d38583556f, State: Initialized, Role: FOLLOWER
I20260926 03:02:45.601261 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:45.600886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:35929)
0926 03:02:45.601076 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:45.601145 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 03:02:45.601255 15415 consensus_queue.cc:260] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
W20260926 03:02:45.601511 15523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:02:45.602687 15415 ts_tablet_manager.cc:1428] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 03:02:45.602770 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:45.602387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:46421)
0926 03:02:45.602534 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:45.602612 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 03:02:45.603231 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 15358
I20260926 03:02:45.603581 15525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:45.603275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:52119)
0926 03:02:45.603418 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:45.603487 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 03:02:45.603909 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41303
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39161
--webserver_port=37077
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:02:45.916070 15527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:45.916378 15527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:45.916541 15527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:45.923627 15527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:45.923772 15527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:45.923938 15527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:02:45.933403 15527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37077
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:45.934652 15527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:45.935993 15527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:45.948731 15532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:45.948773 15535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:45.948691 15533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:45.949483 15527 server_base.cc:1034] running on GCE node
I20260926 03:02:45.950040 15527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:45.950912 15527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:45.952127 15527 hybrid_clock.cc:648] HybridClock initialized: now 1790391765952085 us; error 56 us; skew 500 ppm
I20260926 03:02:45.952425 15527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:45 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:45.955314 15527 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:02:45.956640 15527 webserver.cc:460] Webserver started at http://127.0.97.2:37077/ using document root <none> and password file <none>
I20260926 03:02:45.957366 15527 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:45.957508 15527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:45.962416 15527 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 03:02:45.964900 15542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:45.966147 15527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:02:45.966311 15527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "fbb946f8ffcb452394eb6cff78ccf728"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "6feb70b1e845d2109a39f34556abeadc"
server_key_iv: "a888605fe753471c7732b79385868fd3"
server_key_version: "encryptionkey@0"
I20260926 03:02:45.967252 15527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:45.979950 15527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:45.983598 15527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:45.983920 15527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:45.984813 15527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:45.986111 15527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:45.986230 15527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:45.986359 15527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:45.986450 15527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:46.018147 15527 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41303
I20260926 03:02:46.018226 15654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41303 every 8 connection(s)
I20260926 03:02:46.019546 15527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:02:46.024006 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:46.022350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:47917)
0926 03:02:46.023010 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:46.023217 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":246,"threads_started":1}
W20260926 03:02:46.024788 15655 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39161 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39161: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
I20260926 03:02:46.024827 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 15527
I20260926 03:02:46.025553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46881
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39161
--webserver_port=37141
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:02:46.025707 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:46.025275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:45171)
0926 03:02:46.025492 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:46.025576 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:02:46.026655 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:46.026226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:42049)
0926 03:02:46.026400 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:46.026497 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:02:46.026923 15655 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:02:46.335455 15658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:02:46.335713 15658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:02:46.335801 15658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:02:46.342993 15658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:02:46.343089 15658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:02:46.343237 15658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:02:46.352680 15658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37141
--tserver_master_addrs=127.0.97.62:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:02:46.353984 15658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:02:46.355254 15658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:02:46.368251 15664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:46.368451 15666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:02:46.368750 15663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:02:46.368979 15658 server_base.cc:1034] running on GCE node
I20260926 03:02:46.369704 15658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:02:46.370592 15658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:02:46.371809 15658 hybrid_clock.cc:648] HybridClock initialized: now 1790391766371782 us; error 42 us; skew 500 ppm
I20260926 03:02:46.372116 15658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:02:46 dist-test-slave-fccw krb5kdc[14209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:02:46.375241 15658 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:02:46.376583 15658 webserver.cc:460] Webserver started at http://127.0.97.3:37141/ using document root <none> and password file <none>
I20260926 03:02:46.377377 15658 fs_manager.cc:359] Metadata directory not provided
I20260926 03:02:46.377506 15658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:02:46.382555 15658 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:02:46.385056 15673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:46.386348 15658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:02:46.386520 15658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e"
format_stamp: "Formatted at 2026-09-26 03:02:40 on dist-test-slave-fccw"
server_key: "9b912c59fa722a6e729c57e9c6028a74"
server_key_iv: "76d5cbee2b1f5964b74aedafc13bf3f4"
server_key_version: "encryptionkey@0"
I20260926 03:02:46.387461 15658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:02:46.407441 15658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:02:46.411563 15658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:02:46.411832 15658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:02:46.412781 15658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:02:46.414129 15658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:02:46.414256 15658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:46.414381 15658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:02:46.414467 15658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:02:46.447124 15658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46881
I20260926 03:02:46.447175 15785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46881 every 8 connection(s)
I20260926 03:02:46.448545 15658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:02:46.453052 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:46.451324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:59191)
0926 03:02:46.452005 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:46.452191 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":251,"threads_started":1}
W20260926 03:02:46.453684 15786 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39161 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39161: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
I20260926 03:02:46.454416 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:46.454111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:41191)
0926 03:02:46.454255 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:46.454326 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:02:46.455152 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:46.454880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:55063)
0926 03:02:46.454989 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:46.455056 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 03:02:46.455374 15786 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:02:46.456178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 15658
I20260926 03:02:46.604970 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:46.604249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:60789)
0926 03:02:46.604685 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:46.604775 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":210,"threads_started":1}
I20260926 03:02:47.028093 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:47.027358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:33223)
0926 03:02:47.027779 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:47.027857 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260926 03:02:47.280784 15792 raft_consensus.cc:482] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:47.281138 15792 raft_consensus.cc:504] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:47.282668 15792 leader_election.cc:288] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
Sep 26 03:02:47 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 26 03:02:47 dist-test-slave-fccw krb5kdc[14209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:02:47.305282 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:47.283583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41303 (local address 127.0.97.1:52559)
0926 03:02:47.284121 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:47.284142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:47.284239 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:47.287967 (+ 3728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:47.287971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:47.288634 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:47.289605 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:47.289632 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.291909 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.291914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:47.293428 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:47.293440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.294400 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.294407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:47.294714 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:47.295502 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:47.295518 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:47.298455 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:47.301493 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:47.301502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:47.301516 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:47.301936 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:47.302479 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:47.302484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:47.302487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:47.302740 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:47.303634 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:47.303642 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:47.303989 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 03:02:47.304274 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 03:02:47.304760 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":109,"spinlock_wait_cycles":12160,"thread_start_us":305,"threads_started":1}
I20260926 03:02:47.305335 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:47.284199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52559 (local address 127.0.97.2:41303)
0926 03:02:47.284898 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:47.284914 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:47.284954 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:02:47.287217 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:47.287228 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:47.287671 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:47.287995 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:47.289805 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.291684 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.293567 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.294186 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.294382 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:47.298705 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:47.298712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:47.298723 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:47.298755 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:47.301357 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:47.302078 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:47.302086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:47.302094 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:47.302340 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:47.302861 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:47.302865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:47.302867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:47.303501 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:47.303642 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:47.304538 (+ 896us) server_negotiation.cc:299] Negotiation successful
0926 03:02:47.304910 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":317,"threads_started":1}
I20260926 03:02:47.305500 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:47.283250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49829 (local address 127.0.97.3:46881)
0926 03:02:47.283785 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0926 03:02:47.283800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:02:47.283845 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:02:47.286580 (+ 2735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:02:47.286593 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:02:47.287087 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:02:47.287444 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:02:47.289811 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.291642 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.293573 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.294199 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.294390 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:47.298280 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:02:47.298288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:02:47.298300 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:02:47.298335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:02:47.301351 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:47.302765 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:47.302775 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:47.302784 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:47.302983 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:02:47.303387 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:02:47.303390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:02:47.303392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:02:47.303971 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:02:47.304161 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:02:47.304651 (+ 490us) server_negotiation.cc:299] Negotiation successful
0926 03:02:47.305053 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":264,"threads_started":1}
I20260926 03:02:47.305377 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:47.282872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46881 (local address 127.0.97.1:49829)
0926 03:02:47.283381 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:47.283411 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:02:47.284065 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:02:47.287565 (+ 3500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:02:47.287573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:02:47.288499 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:02:47.289605 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:47.289631 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.291882 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.291886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:47.293434 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:02:47.293445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:02:47.294422 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:02:47.294429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:02:47.294715 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:02:47.295441 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:02:47.295470 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:02:47.298016 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:02:47.302092 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:47.302097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:47.302100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:47.302611 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:47.303111 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:02:47.303115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:02:47.303118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:02:47.303278 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:02:47.304106 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:02:47.304109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:02:47.304228 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 03:02:47.304382 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 03:02:47.304910 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":380,"spinlock_wait_cycles":16768,"thread_start_us":278,"threads_started":1}
W20260926 03:02:47.308943 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
W20260926 03:02:47.309315 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:47.309490 15410 leader_election.cc:302] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f; no voters: fa1ab79108ff4d4bbb5ae3f3bf34e66e, fbb946f8ffcb452394eb6cff78ccf728
I20260926 03:02:47.309857 15792 raft_consensus.cc:2726] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:02:47.456692 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:47.455909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:45051)
0926 03:02:47.456342 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:47.456464 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260926 03:02:47.606120 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:47.605743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:54339)
0926 03:02:47.605884 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:47.605976 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 03:02:48.029587 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:48.028884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:33097)
0926 03:02:48.029344 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:48.029415 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":277,"threads_started":1}
I20260926 03:02:48.458110 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:48.457459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:49719)
0926 03:02:48.457881 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:48.457957 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":224,"threads_started":1}
I20260926 03:02:48.607669 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:48.606934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:40849)
0926 03:02:48.607415 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:48.607490 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":227,"threads_started":1}
I20260926 03:02:49.031061 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:49.030427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:44489)
0926 03:02:49.030827 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:49.030902 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":217,"threads_started":1}
I20260926 03:02:49.459985 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:49.459194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:47199)
0926 03:02:49.459711 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:49.459819 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":295,"threads_started":1}
I20260926 03:02:49.518257 15803 raft_consensus.cc:482] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:49.518483 15803 raft_consensus.cc:504] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:49.519099 15803 leader_election.cc:288] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
W20260926 03:02:49.520125 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
W20260926 03:02:49.520385 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:49.520495 15410 leader_election.cc:302] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f; no voters: fa1ab79108ff4d4bbb5ae3f3bf34e66e, fbb946f8ffcb452394eb6cff78ccf728
I20260926 03:02:49.520902 15803 raft_consensus.cc:2726] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:02:49.609326 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:49.608540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:37943)
0926 03:02:49.608976 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:49.609119 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":206,"threads_started":1}
I20260926 03:02:50.032627 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:50.031960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:43619)
0926 03:02:50.032386 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:50.032456 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
I20260926 03:02:50.461582 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:50.460904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:57077)
0926 03:02:50.461328 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:50.461402 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260926 03:02:50.611013 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:50.610302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:51101)
0926 03:02:50.610752 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:50.610837 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":239,"threads_started":1}
I20260926 03:02:51.034216 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:51.033589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:41161)
0926 03:02:51.033995 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:51.034068 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":230,"threads_started":1}
I20260926 03:02:51.463171 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:51.462459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:42727)
0926 03:02:51.462896 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:51.462989 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":215,"threads_started":1}
I20260926 03:02:51.612524 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:51.611865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:36885)
0926 03:02:51.612294 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:51.612373 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":221,"threads_started":1}
I20260926 03:02:51.707548 15811 raft_consensus.cc:482] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:51.707773 15811 raft_consensus.cc:504] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:51.708420 15811 leader_election.cc:288] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
W20260926 03:02:51.709388 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
W20260926 03:02:51.709619 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:51.709743 15410 leader_election.cc:302] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f; no voters: fa1ab79108ff4d4bbb5ae3f3bf34e66e, fbb946f8ffcb452394eb6cff78ccf728
I20260926 03:02:51.709992 15811 raft_consensus.cc:2726] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:02:52.035781 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:52.035130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:37033)
0926 03:02:52.035527 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:52.035599 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":212,"threads_started":1}
I20260926 03:02:52.464747 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:52.464059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:54113)
0926 03:02:52.464477 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:52.464553 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260926 03:02:52.614027 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:52.613366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:38595)
0926 03:02:52.613758 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:52.613840 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
I20260926 03:02:53.037449 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:53.036644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:37639)
0926 03:02:53.037163 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:53.037264 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":212,"threads_started":1}
I20260926 03:02:53.466432 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:53.465630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:51919)
0926 03:02:53.466107 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:53.466189 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":215,"threads_started":1}
I20260926 03:02:53.615477 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:53.614810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:40089)
0926 03:02:53.615238 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:53.615315 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260926 03:02:54.038978 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:54.038298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:49373)
0926 03:02:54.038728 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:54.038808 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260926 03:02:54.467959 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:54.467277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:52349)
0926 03:02:54.467696 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:54.467770 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260926 03:02:54.616950 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:54.616291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:37531)
0926 03:02:54.616697 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:54.616770 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":215,"threads_started":1}
I20260926 03:02:55.041060 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:55.040098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:37783)
0926 03:02:55.040645 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:55.040761 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260926 03:02:55.469833 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:55.468983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:33671)
0926 03:02:55.469526 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:55.469625 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":214,"threads_started":1}
I20260926 03:02:55.618637 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:55.617957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:50609)
0926 03:02:55.618395 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:55.618473 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":213,"threads_started":1}
I20260926 03:02:55.796742 15824 raft_consensus.cc:482] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:55.796967 15824 raft_consensus.cc:504] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:55.797680 15824 leader_election.cc:288] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
W20260926 03:02:55.798632 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
W20260926 03:02:55.798935 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:55.799047 15410 leader_election.cc:302] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f; no voters: fa1ab79108ff4d4bbb5ae3f3bf34e66e, fbb946f8ffcb452394eb6cff78ccf728
I20260926 03:02:55.799355 15824 raft_consensus.cc:2726] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:02:56.042691 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:56.041978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:57017)
0926 03:02:56.042420 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:56.042516 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":239,"threads_started":1}
I20260926 03:02:56.471452 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:56.470661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:36941)
0926 03:02:56.471110 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:56.471224 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":252,"threads_started":1}
I20260926 03:02:56.620221 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:56.619529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:41673)
0926 03:02:56.619982 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:56.620057 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260926 03:02:57.044363 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:57.043531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:35649)
0926 03:02:57.044122 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:57.044200 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":359,"threads_started":1}
I20260926 03:02:57.473045 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:57.472275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:51095)
0926 03:02:57.472758 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:57.472837 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":252,"threads_started":1}
I20260926 03:02:57.621865 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:57.621130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:55319)
0926 03:02:57.621589 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:57.621671 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":251,"threads_started":1}
I20260926 03:02:58.045976 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:58.045245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:51213)
0926 03:02:58.045726 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:58.045801 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":237,"threads_started":1}
I20260926 03:02:58.359992 15832 raft_consensus.cc:482] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:02:58.360231 15832 raft_consensus.cc:504] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0880de7f65149e3ac55a9d38583556f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46139 } } peers { permanent_uuid: "fa1ab79108ff4d4bbb5ae3f3bf34e66e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46881 } } peers { permanent_uuid: "fbb946f8ffcb452394eb6cff78ccf728" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41303 } }
I20260926 03:02:58.360841 15832 leader_election.cc:288] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881), fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303)
W20260926 03:02:58.362056 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa1ab79108ff4d4bbb5ae3f3bf34e66e (127.0.97.3:46881): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
W20260926 03:02:58.362334 15410 leader_election.cc:341] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbb946f8ffcb452394eb6cff78ccf728 (127.0.97.2:41303): Not found: Tablet not found: 6d5a3a5dd1994452b921c2620804b6b7
I20260926 03:02:58.362466 15410 leader_election.cc:302] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0880de7f65149e3ac55a9d38583556f; no voters: fa1ab79108ff4d4bbb5ae3f3bf34e66e, fbb946f8ffcb452394eb6cff78ccf728
I20260926 03:02:58.362742 15832 raft_consensus.cc:2726] T 6d5a3a5dd1994452b921c2620804b6b7 P e0880de7f65149e3ac55a9d38583556f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:02:58.474520 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:58.473859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:44803)
0926 03:02:58.474284 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:58.474356 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":213,"threads_started":1}
I20260926 03:02:58.623431 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:58.622726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:34821)
0926 03:02:58.623154 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:58.623240 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":221,"threads_started":1}
I20260926 03:02:59.047694 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:59.047009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:52399)
0926 03:02:59.047454 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:59.047531 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
I20260926 03:02:59.476020 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:59.475358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:51183)
0926 03:02:59.475787 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:59.475861 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":220,"threads_started":1}
I20260926 03:02:59.625298 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:02:59.624308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:57747)
0926 03:02:59.624876 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 03:02:59.624988 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":295,"threads_started":1}
I20260926 03:03:00.049295 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:00.048518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:40151)
0926 03:03:00.048927 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:00.049074 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":211,"threads_started":1}
I20260926 03:03:00.477622 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:00.476832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:49657)
0926 03:03:00.477314 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:00.477407 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":291,"threads_started":1}
I20260926 03:03:00.627321 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:00.626522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.1:45487)
0926 03:03:00.627046 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:00.627147 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":276,"threads_started":1}
I20260926 03:03:01.051002 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:01.050360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.2:36219)
0926 03:03:01.050769 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:01.050850 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
I20260926 03:03:01.467566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 15358
I20260926 03:03:01.479296 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:01.478632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39161 (local address 127.0.97.3:36109)
0926 03:03:01.479051 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:01.479144 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
I20260926 03:03:01.488849 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:01.468631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.97.1:46139)
0926 03:03:01.469248 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:01.469258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:01.469291 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:03:01.469379 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:01.469390 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:01.469682 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:01.469939 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:01.471843 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:01.472745 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:01.474018 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:01.474551 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:01.474684 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:01.483448 (+ 8764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:01.483460 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:01.483475 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:01.483506 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:01.486115 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:01.486817 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:01.486824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:01.486827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:01.486949 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:01.487322 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:01.487334 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:01.487336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:01.487666 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:01.487980 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:01.488263 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 03:03:01.488528 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":279,"threads_started":1}
I20260926 03:03:01.490164 15518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:01.578555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 15358
I20260926 03:03:01.596081 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 15527
I20260926 03:03:01.608718 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:01.596897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.97.2:41303)
0926 03:03:01.597396 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:01.597401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:01.597417 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:03:01.597472 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:01.597476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:01.597533 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:01.597624 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:01.598931 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:01.599712 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:01.600894 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:01.601356 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:01.601447 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:01.603845 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:01.603850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:01.603853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:01.603916 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:01.606670 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:01.607251 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:01.607254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:01.607257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:01.607369 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:01.607668 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:01.607671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:01.607673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:01.607913 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:01.608099 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:01.608272 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 03:03:01.608434 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":306,"threads_started":1}
I20260926 03:03:01.609665 15650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:01.692831 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 15527
I20260926 03:03:01.710469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 15658
I20260926 03:03:01.722975 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:01.711317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.0.97.3:46881)
0926 03:03:01.711885 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:01.711890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:01.711906 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:03:01.711962 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:01.711965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:01.712032 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:01.712124 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:01.713448 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:01.714220 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:01.715388 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:01.715798 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:01.715892 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:01.718401 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:01.718405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:01.718409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:01.718437 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:01.720809 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:01.721454 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:01.721457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:01.721460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:01.721590 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:01.721892 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:01.721895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:01.721897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:01.722116 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:01.722319 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:01.722539 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 03:03:01.722687 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":309,"threads_started":1}
I20260926 03:03:01.723721 15781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:01.805826 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 15658
2026-09-26T03:03:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24153 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:03:01 dist-test-slave-fccw krb5kdc[15851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:03:01 dist-test-slave-fccw krb5kdc[15851](info): set up 2 sockets
Sep 26 03:03:01 dist-test-slave-fccw krb5kdc[15851](info): commencing operation
krb5kdc: starting...
W20260926 03:03:02.889551 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:03:03 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:03:03.118001 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37913
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:03:03.449780 15864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:03.450125 15864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:03.450284 15864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:03.457199 15864 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:03:03.457281 15864 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:03:03.457342 15864 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:03:03.457442 15864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:03:03.457530 15864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:03.457646 15864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:03.457723 15864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:03:03.457799 15864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:03:03.467315 15864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:03.468518 15864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:03.469825 15864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:03:03.482312 15864 server_base.cc:1034] running on GCE node
W20260926 03:03:03.482395 15872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:03.482307 15870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:03.482347 15869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:03.483455 15864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:03.484797 15864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:03.486035 15864 hybrid_clock.cc:648] HybridClock initialized: now 1790391783485960 us; error 52 us; skew 500 ppm
I20260926 03:03:03.486318 15864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:03 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:03.489436 15864 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:03:03.490782 15864 webserver.cc:460] Webserver started at http://127.0.97.62:44347/ using document root <none> and password file <none>
I20260926 03:03:03.491418 15864 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:03.491500 15864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:03.491734 15864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:03:03.494691 15864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5"
format_stamp: "Formatted at 2026-09-26 03:03:03 on dist-test-slave-fccw"
server_key: "87d1b85414785bf33ecdf6c07aee4494"
server_key_iv: "f32f48af6ef3af7af3618cdca23557e6"
server_key_version: "encryptionkey@0"
I20260926 03:03:03.495393 15864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5"
format_stamp: "Formatted at 2026-09-26 03:03:03 on dist-test-slave-fccw"
server_key: "87d1b85414785bf33ecdf6c07aee4494"
server_key_iv: "f32f48af6ef3af7af3618cdca23557e6"
server_key_version: "encryptionkey@0"
I20260926 03:03:03.500332 15864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:03:03.504034 15879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:03.505615 15864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:03:03.505774 15864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5"
format_stamp: "Formatted at 2026-09-26 03:03:03 on dist-test-slave-fccw"
server_key: "87d1b85414785bf33ecdf6c07aee4494"
server_key_iv: "f32f48af6ef3af7af3618cdca23557e6"
server_key_version: "encryptionkey@0"
I20260926 03:03:03.505944 15864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:03.531579 15864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:03.535812 15864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:03.536047 15864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:03.554140 15864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37913
I20260926 03:03:03.554157 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37913 every 8 connection(s)
I20260926 03:03:03.555500 15864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:03:03.559087 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:03:03.562109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 15864
I20260926 03:03:03.562342 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:03:03.563162 388 external_mini_cluster.cc:1428] Setting key adfb927e3e5271d914e7dcea50c46ebe
I20260926 03:03:03.571776 15931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Bootstrap starting.
I20260926 03:03:03.575572 15931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Neither blocks nor log segments found. Creating new log.
I20260926 03:03:03.576875 15931 log.cc:824] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:03:03 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:03:03.579747 15931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: No bootstrap required, opened a new log
I20260926 03:03:03.586813 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:03.565313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.97.62:37913)
0926 03:03:03.566067 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:03.566079 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:03.566119 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:03:03.567795 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:03.567818 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:03.568210 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:03.568539 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:03.570446 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:03.571714 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:03.573676 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:03.574317 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:03.574451 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:03.579343 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:03.579350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:03.579364 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:03.579409 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:03.582647 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:03.583447 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:03.583453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:03.583459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:03.583631 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:03.584064 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:03.584073 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:03.584075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:03.584594 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:03.584925 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:03.585394 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 03:03:03.585803 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":361,"threads_started":1}
I20260926 03:03:03.587775 15931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:03.588141 15931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:03:03.588311 15931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5244f0a890cb452ea1e2ca44ef3d3eb5, State: Initialized, Role: FOLLOWER
I20260926 03:03:03.588868 15931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:03.589186 15931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:03:03.589346 15931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:03:03.589538 15931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:03:03.592044 15931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:03.592496 15931 leader_election.cc:302] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5244f0a890cb452ea1e2ca44ef3d3eb5; no voters:
I20260926 03:03:03.593082 15931 leader_election.cc:288] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:03:03.593233 15936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:03:03.593894 15936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 5244f0a890cb452ea1e2ca44ef3d3eb5, State: Running, Role: LEADER
I20260926 03:03:03.594440 15936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:03.595093 15931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:03:03.596984 15938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5244f0a890cb452ea1e2ca44ef3d3eb5. Latest consensus state: current_term: 1 leader_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } } }
I20260926 03:03:03.597616 15938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:03:03.597585 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } } }
I20260926 03:03:03.598258 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:03:03.602051 15942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:03:03.607756 15942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:03:03.616351 15942 catalog_manager.cc:1347] Generated new cluster ID: a65fcab1744e41e79a544b3965e06545
I20260926 03:03:03.616526 15942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:03:03.627712 15942 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:03:03.628917 15942 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:03:03.642170 15942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Generated new TSK 0
I20260926 03:03:03.643126 15942 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:03:03.701030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37913
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:03:04.020000 15957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:04.020315 15957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:04.020474 15957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:04.027535 15957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:04.027679 15957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:04.027848 15957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:03:04.037248 15957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:04.038524 15957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:04.040016 15957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:04.052703 15963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:04.052660 15965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:04.052734 15962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:04.053575 15957 server_base.cc:1034] running on GCE node
I20260926 03:03:04.054111 15957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:04.054951 15957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:04.056147 15957 hybrid_clock.cc:648] HybridClock initialized: now 1790391784056102 us; error 44 us; skew 500 ppm
I20260926 03:03:04.056454 15957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:04 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:04.059549 15957 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:03:04.060911 15957 webserver.cc:460] Webserver started at http://127.0.97.1:41815/ using document root <none> and password file <none>
I20260926 03:03:04.061594 15957 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:04.061736 15957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:04.062054 15957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:03:04.064927 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "fb3741b86c3b4020a6265d8f2553024e"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "5e50ac8254bdcaba279093e43db605a8"
server_key_iv: "a61afcbb37d1db970eb5eb28093af889"
server_key_version: "encryptionkey@0"
I20260926 03:03:04.065693 15957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fb3741b86c3b4020a6265d8f2553024e"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "5e50ac8254bdcaba279093e43db605a8"
server_key_iv: "a61afcbb37d1db970eb5eb28093af889"
server_key_version: "encryptionkey@0"
I20260926 03:03:04.070542 15957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:03:04.073860 15972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:04.075102 15957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:03:04.075273 15957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "fb3741b86c3b4020a6265d8f2553024e"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "5e50ac8254bdcaba279093e43db605a8"
server_key_iv: "a61afcbb37d1db970eb5eb28093af889"
server_key_version: "encryptionkey@0"
I20260926 03:03:04.075469 15957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:04.096489 15957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:04.100415 15957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:04.100667 15957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:04.101639 15957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:04.102932 15957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:04.103001 15957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:04.103175 15957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:04.103248 15957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:04.135350 15957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41445
I20260926 03:03:04.135407 16084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41445 every 8 connection(s)
I20260926 03:03:04.136595 15957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:03:04.143823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 15957
I20260926 03:03:04.144039 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:03:04.144832 388 external_mini_cluster.cc:1428] Setting key 747a86a87e97e0900dbab9ce179c2f82
Sep 26 03:03:04 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:03:04.156103 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:04.139117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59133 (local address 127.0.97.62:37913)
0926 03:03:04.139557 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:04.139562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:04.140373 (+ 811us) server_negotiation.cc:407] Connection header received
0926 03:03:04.142454 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:04.142460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:04.142545 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:04.142658 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:04.144258 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.145151 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.146839 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.147237 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.147366 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:04.150594 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:04.150599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:04.150602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:04.150632 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:04.153180 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:04.153859 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:04.153866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:04.153870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:04.154007 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:04.154455 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:04.154461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:04.154464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:04.154720 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:04.154873 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:04.155520 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 03:03:04.155737 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":245,"threads_started":1}
I20260926 03:03:04.157095 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:04.139528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:59133)
0926 03:03:04.140209 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:04.140250 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:04.142194 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:04.142892 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:04.142899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:04.143453 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:03:04.144061 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:04.144102 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.145284 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.145289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:04.146676 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:04.146687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.147454 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.147461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:04.147675 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:04.148346 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:03:04.148370 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:03:04.150385 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:03:04.153309 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:04.153317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:04.153331 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:04.153739 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:04.154143 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:04.154149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:04.154152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:04.154338 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:04.154854 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:03:04.154862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:03:04.155168 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 03:03:04.155422 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:03:04.155931 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":279,"threads_started":1}
I20260926 03:03:04.158515 16085 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37913
I20260926 03:03:04.158828 16085 heartbeater.cc:464] Registering TS with master...
I20260926 03:03:04.159657 16085 heartbeater.cc:511] Master 127.0.97.62:37913 requested a full tablet report, sending...
I20260926 03:03:04.161767 15896 ts_manager.cc:194] Registered new tserver with Master: fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445)
I20260926 03:03:04.163141 15896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59133
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:03:04.199630 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37913
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:03:04.515513 16091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:04.515792 16091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:04.515895 16091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:04.523088 16091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:04.523193 16091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:04.523303 16091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:03:04.532718 16091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:04.534013 16091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:04.535243 16091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:04.547894 16096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:04.547928 16099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:04.547854 16097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:04.548744 16091 server_base.cc:1034] running on GCE node
I20260926 03:03:04.549470 16091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:04.550348 16091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:04.551568 16091 hybrid_clock.cc:648] HybridClock initialized: now 1790391784551527 us; error 35 us; skew 500 ppm
I20260926 03:03:04.551919 16091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:04 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:04.555531 16091 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:03:04.566113 16091 webserver.cc:460] Webserver started at http://127.0.97.2:33793/ using document root <none> and password file <none>
I20260926 03:03:04.566821 16091 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:04.566954 16091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:04.567270 16091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:03:04.570231 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "98a2feaa704638c9d907875b0e4b16ba"
server_key_iv: "311eacb1d951af0da8dd3bc0034be445"
server_key_version: "encryptionkey@0"
I20260926 03:03:04.570971 16091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "98a2feaa704638c9d907875b0e4b16ba"
server_key_iv: "311eacb1d951af0da8dd3bc0034be445"
server_key_version: "encryptionkey@0"
I20260926 03:03:04.575805 16091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:03:04.579210 16106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:04.580528 16091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:03:04.580686 16091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "98a2feaa704638c9d907875b0e4b16ba"
server_key_iv: "311eacb1d951af0da8dd3bc0034be445"
server_key_version: "encryptionkey@0"
I20260926 03:03:04.580847 16091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:04.599354 16091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:04.603175 16091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:04.603461 16091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:04.604362 16091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:04.605600 16091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:04.605669 16091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:04.605835 16091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:04.605883 16091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:04.638023 16091 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43063
I20260926 03:03:04.638060 16218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43063 every 8 connection(s)
I20260926 03:03:04.639256 16091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:03:04.640906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16091
I20260926 03:03:04.641144 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:03:04.641734 388 external_mini_cluster.cc:1428] Setting key b288d4805a6c12e3f32dad7124613c90
Sep 26 03:03:04 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:03:04.663793 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:04.642939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40923 (local address 127.0.97.62:37913)
0926 03:03:04.643196 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:04.643202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:04.644551 (+ 1349us) server_negotiation.cc:407] Connection header received
0926 03:03:04.647448 (+ 2897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:04.647452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:04.647548 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:04.647662 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:04.649639 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.650299 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.652802 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.653434 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.653580 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:04.657016 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:04.657025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:04.657034 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:04.657077 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:04.660651 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:04.661356 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:04.661364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:04.661370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:04.661524 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:04.661912 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:04.661919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:04.661925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:04.662293 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:04.662441 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:04.663058 (+ 617us) server_negotiation.cc:299] Negotiation successful
0926 03:03:04.663327 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:03:04.664361 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:04.643542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:40923)
0926 03:03:04.644371 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:04.644428 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:04.647179 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:04.647906 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:04.647918 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:04.648653 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:03:04.649458 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:04.649484 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.650448 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.650456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:04.652646 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:04.652664 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:04.653637 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:04.653642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:04.653836 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:04.654505 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:03:04.654530 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:03:04.656771 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:03:04.660798 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:04.660804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:04.660815 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:04.661247 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:04.661630 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:04.661633 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:04.661635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:04.661794 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:04.662404 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:03:04.662417 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:03:04.662703 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 03:03:04.662910 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 03:03:04.663255 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":318,"threads_started":1}
I20260926 03:03:04.665719 16219 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37913
I20260926 03:03:04.666077 16219 heartbeater.cc:464] Registering TS with master...
I20260926 03:03:04.666996 16219 heartbeater.cc:511] Master 127.0.97.62:37913 requested a full tablet report, sending...
I20260926 03:03:04.668597 15896 ts_manager.cc:194] Registered new tserver with Master: 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260926 03:03:04.669478 15896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:03:04.697438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37913
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:03:05.007718 16224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:05.007978 16224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:05.008069 16224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:05.015162 16224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:05.015251 16224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:05.015388 16224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:03:05.024559 16224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:05.025708 16224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:05.026933 16224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:05.039288 16229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:05.039391 16230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:05.039354 16232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:05.040417 16224 server_base.cc:1034] running on GCE node
I20260926 03:03:05.040908 16224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:05.041797 16224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:05.043011 16224 hybrid_clock.cc:648] HybridClock initialized: now 1790391785042963 us; error 27 us; skew 500 ppm
I20260926 03:03:05.043345 16224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:05 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:05.046622 16224 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:03:05.048018 16224 webserver.cc:460] Webserver started at http://127.0.97.3:40975/ using document root <none> and password file <none>
I20260926 03:03:05.048818 16224 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:05.048952 16224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:05.049289 16224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:03:05.052081 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "7133119c685e47ed82ac25004c5d4265"
format_stamp: "Formatted at 2026-09-26 03:03:05 on dist-test-slave-fccw"
server_key: "2c38527667367f9eb2941a56eb961563"
server_key_iv: "0f37e7a9ff6f74c5f1e12e04d287c34f"
server_key_version: "encryptionkey@0"
I20260926 03:03:05.052799 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7133119c685e47ed82ac25004c5d4265"
format_stamp: "Formatted at 2026-09-26 03:03:05 on dist-test-slave-fccw"
server_key: "2c38527667367f9eb2941a56eb961563"
server_key_iv: "0f37e7a9ff6f74c5f1e12e04d287c34f"
server_key_version: "encryptionkey@0"
I20260926 03:03:05.057804 16224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 03:03:05.061126 16239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:05.062445 16224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:03:05.062623 16224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "7133119c685e47ed82ac25004c5d4265"
format_stamp: "Formatted at 2026-09-26 03:03:05 on dist-test-slave-fccw"
server_key: "2c38527667367f9eb2941a56eb961563"
server_key_iv: "0f37e7a9ff6f74c5f1e12e04d287c34f"
server_key_version: "encryptionkey@0"
I20260926 03:03:05.062824 16224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:05.091236 16224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:05.096685 16224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:05.097034 16224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:05.098318 16224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:05.099968 16224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:05.100055 16224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:05.100220 16224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:05.100296 16224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:05.133026 16224 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40987
I20260926 03:03:05.133088 16351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40987 every 8 connection(s)
I20260926 03:03:05.134276 16224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:03:05.137739 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16224
I20260926 03:03:05.137925 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:03:05.138538 388 external_mini_cluster.cc:1428] Setting key 0612785c4d1c55b498be307cc1bc3f49
Sep 26 03:03:05 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:03:05.154943 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.137032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36839 (local address 127.0.97.62:37913)
0926 03:03:05.137192 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.137196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.138824 (+ 1628us) server_negotiation.cc:407] Connection header received
0926 03:03:05.141439 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.141444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.141547 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.141669 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:05.143209 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.143968 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.145530 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.145891 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.145980 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.149402 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:05.149406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:05.149410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:05.149448 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:05.151720 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.152487 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.152492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.152495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.152651 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.153127 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.153131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.153135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.153491 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:05.153636 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.154480 (+ 844us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.154645 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:03:05.155905 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.137611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:36839)
0926 03:03:05.138429 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:05.138572 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:05.141191 (+ 2619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:05.141961 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:05.141975 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:05.142508 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:03:05.143066 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.143082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.144114 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.144118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.145415 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.145427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.146164 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.146169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.146382 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.147005 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:03:05.147025 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:03:05.149211 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:03:05.151847 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:05.151853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:05.151863 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:05.152336 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:05.152757 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:05.152761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:05.152764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:05.152994 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:05.153631 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:03:05.153641 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:03:05.154031 (+ 390us) client_negotiation.cc:769] Sending connection context
0926 03:03:05.154372 (+ 341us) client_negotiation.cc:240] Negotiation successful
0926 03:03:05.154757 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":335,"threads_started":1}
I20260926 03:03:05.157362 16352 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37913
I20260926 03:03:05.157655 16352 heartbeater.cc:464] Registering TS with master...
I20260926 03:03:05.158335 16352 heartbeater.cc:511] Master 127.0.97.62:37913 requested a full tablet report, sending...
I20260926 03:03:05.159781 15896 ts_manager.cc:194] Registered new tserver with Master: 7133119c685e47ed82ac25004c5d4265 (127.0.97.3:40987)
I20260926 03:03:05.160557 15896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36839
I20260926 03:03:05.166563 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:03:05.166637 16085 heartbeater.cc:503] Master 127.0.97.62:37913 was elected leader, sending a full tablet report...
I20260926 03:03:05.183591 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.171417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.0.97.62:37913)
0926 03:03:05.171626 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.171635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.172052 (+ 417us) server_negotiation.cc:407] Connection header received
0926 03:03:05.172170 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.172174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.172229 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.172307 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:05.173748 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.174425 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.175773 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.176153 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.176240 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.178599 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:05.178604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:05.178607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:05.178634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:05.181243 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.181919 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.181922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.181925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.182032 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.182323 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.182325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.182327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.182607 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:05.182775 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.183068 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.183217 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:03:05.187568 15896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:03:05.194474 15896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:03:05.223714 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.206112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40987 (local address 127.0.0.1:48122)
0926 03:03:05.206977 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:05.207023 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:05.207218 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:05.208628 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:05.208639 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:05.208681 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:05.210484 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.210504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.214581 (+ 4077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.214590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.218915 (+ 4325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.218935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.221702 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.221711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.221988 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.222059 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 03:03:05.223121 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0926 03:03:05.223318 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":346,"threads_started":1}
I20260926 03:03:05.223869 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.206398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.0.97.3:40987)
0926 03:03:05.207519 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.207531 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.207572 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:03:05.207697 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.207708 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.207994 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.208219 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:05.210745 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.214376 (+ 3631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.219142 (+ 4766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.221072 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.221226 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.221527 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.223348 (+ 1821us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.223515 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":725,"threads_started":1}
I20260926 03:03:05.228518 16287 tablet_service.cc:1467] Processing CreateTablet for tablet 9913752e47fd4df6a1337903aada59f5 (DEFAULT_TABLE table=test-table [id=1be46c8552224196a15bbc861870f223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:03:05.230087 16287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9913752e47fd4df6a1337903aada59f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:03:05.231688 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.207848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41445 (local address 127.0.0.1:58080)
0926 03:03:05.224340 (+ 16492us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:05.224360 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:05.224459 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:05.226182 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:05.226185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:05.226204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:05.226315 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.226322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.227955 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.227963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.229642 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.229653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.230898 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.230904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.231113 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.231150 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:03:05.231259 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:03:05.231384 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16386,"thread_start_us":17765,"threads_started":1}
I20260926 03:03:05.231797 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.208408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.0.97.1:41445)
0926 03:03:05.224660 (+ 16252us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.224670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.224699 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:03:05.224799 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.224807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.225098 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.225284 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:05.226467 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.227811 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.229787 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.230744 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.230885 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.231104 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.231301 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.231489 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16068,"thread_start_us":15890,"threads_started":1}
I20260926 03:03:05.233624 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.207292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43063 (local address 127.0.0.1:50584)
0926 03:03:05.223127 (+ 15835us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:05.223149 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:05.223277 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:05.226702 (+ 3425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:05.226706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:05.226732 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:05.227115 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.227126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.228949 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.228954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.231011 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.231025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.232823 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.232830 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.233049 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.233088 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:03:05.233182 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:03:05.233310 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15674,"thread_start_us":15526,"threads_started":1}
I20260926 03:03:05.233740 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.207720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.97.2:43063)
0926 03:03:05.226139 (+ 18419us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.226147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.226174 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:03:05.226270 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.226276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.226497 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.226699 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:05.227296 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.228791 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.231190 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.232592 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.232745 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.232933 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.233253 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.233415 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18278,"thread_start_us":18170,"threads_started":1}
I20260926 03:03:05.235594 16020 tablet_service.cc:1467] Processing CreateTablet for tablet 9913752e47fd4df6a1337903aada59f5 (DEFAULT_TABLE table=test-table [id=1be46c8552224196a15bbc861870f223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:03:05.237221 16020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9913752e47fd4df6a1337903aada59f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:03:05.237267 16154 tablet_service.cc:1467] Processing CreateTablet for tablet 9913752e47fd4df6a1337903aada59f5 (DEFAULT_TABLE table=test-table [id=1be46c8552224196a15bbc861870f223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:03:05.238534 16154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9913752e47fd4df6a1337903aada59f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:03:05.245100 16370 tablet_bootstrap.cc:492] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Bootstrap starting.
I20260926 03:03:05.248885 16371 tablet_bootstrap.cc:492] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Bootstrap starting.
I20260926 03:03:05.249023 16370 tablet_bootstrap.cc:654] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Neither blocks nor log segments found. Creating new log.
I20260926 03:03:05.250209 16370 log.cc:824] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:05.252219 16371 tablet_bootstrap.cc:654] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Neither blocks nor log segments found. Creating new log.
I20260926 03:03:05.252915 16370 tablet_bootstrap.cc:492] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: No bootstrap required, opened a new log
I20260926 03:03:05.253165 16372 tablet_bootstrap.cc:492] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Bootstrap starting.
I20260926 03:03:05.253140 16371 log.cc:824] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:05.253201 16370 ts_tablet_manager.cc:1397] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 03:03:05.256014 16371 tablet_bootstrap.cc:492] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: No bootstrap required, opened a new log
I20260926 03:03:05.256247 16372 tablet_bootstrap.cc:654] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Neither blocks nor log segments found. Creating new log.
I20260926 03:03:05.256286 16371 ts_tablet_manager.cc:1397] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 03:03:05.257408 16372 log.cc:824] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:05.260327 16372 tablet_bootstrap.cc:492] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: No bootstrap required, opened a new log
I20260926 03:03:05.260582 16372 ts_tablet_manager.cc:1397] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 03:03:05.261586 16370 raft_consensus.cc:348] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.261917 16370 raft_consensus.cc:374] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:03:05.262023 16370 raft_consensus.cc:729] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133119c685e47ed82ac25004c5d4265, State: Initialized, Role: FOLLOWER
I20260926 03:03:05.262637 16370 consensus_queue.cc:260] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.263667 16352 heartbeater.cc:503] Master 127.0.97.62:37913 was elected leader, sending a full tablet report...
I20260926 03:03:05.265429 16370 ts_tablet_manager.cc:1428] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260926 03:03:05.266443 16371 raft_consensus.cc:348] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.266911 16371 raft_consensus.cc:374] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:03:05.267015 16371 raft_consensus.cc:729] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3741b86c3b4020a6265d8f2553024e, State: Initialized, Role: FOLLOWER
I20260926 03:03:05.267656 16371 consensus_queue.cc:260] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.270650 16371 ts_tablet_manager.cc:1428] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260926 03:03:05.271428 16372 raft_consensus.cc:348] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.271760 16372 raft_consensus.cc:374] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:03:05.271860 16372 raft_consensus.cc:729] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916eb5d32cbb4c9a9a3e233d4fabcfec, State: Initialized, Role: FOLLOWER
I20260926 03:03:05.272324 16372 consensus_queue.cc:260] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.273488 16219 heartbeater.cc:503] Master 127.0.97.62:37913 was elected leader, sending a full tablet report...
I20260926 03:03:05.273938 16372 ts_tablet_manager.cc:1428] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 03:03:05.278285 16376 raft_consensus.cc:482] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:03:05.278568 16376 raft_consensus.cc:504] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.280220 16376 leader_election.cc:288] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063), fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445)
I20260926 03:03:05.286199 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.280720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42853 (local address 127.0.97.1:41445)
0926 03:03:05.280917 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.280925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.281017 (+ 92us) server_negotiation.cc:407] Connection header received
0926 03:03:05.281265 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.281268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.281321 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.281424 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:05.281776 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.282437 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.284056 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.285244 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.285370 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.285488 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.285849 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.285986 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:03:05.286432 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.280512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41445 (local address 127.0.97.3:42853)
0926 03:03:05.280921 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:05.280941 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:05.281122 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:05.281441 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:05.281446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:05.281473 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:05.281650 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.281658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.282558 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.282565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.283884 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.283895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.285552 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.285557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.285696 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.285720 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:03:05.285793 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 03:03:05.285931 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260926 03:03:05.286738 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.280470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39131 (local address 127.0.97.2:43063)
0926 03:03:05.280669 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.280678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.281371 (+ 693us) server_negotiation.cc:407] Connection header received
0926 03:03:05.281541 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.281545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.281602 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.281693 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:05.282164 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.282795 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.284586 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.285760 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.285908 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.286015 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.286339 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.286468 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:03:05.286971 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.280527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43063 (local address 127.0.97.3:39131)
0926 03:03:05.281265 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:05.281287 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:05.281418 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:05.281705 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:05.281710 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:05.281732 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:05.282027 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.282035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.282951 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.282956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.284408 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:05.284419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.285996 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.286001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:05.286168 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.286197 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:03:05.286300 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 03:03:05.286411 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":41,"thread_start_us":356,"threads_started":1}
I20260926 03:03:05.287314 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9913752e47fd4df6a1337903aada59f5" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3741b86c3b4020a6265d8f2553024e" is_pre_election: true
I20260926 03:03:05.287801 16040 raft_consensus.cc:2454] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 0.
I20260926 03:03:05.287873 16174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9913752e47fd4df6a1337903aada59f5" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" is_pre_election: true
I20260926 03:03:05.288326 16174 raft_consensus.cc:2454] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 0.
I20260926 03:03:05.288378 16241 leader_election.cc:302] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133119c685e47ed82ac25004c5d4265, fb3741b86c3b4020a6265d8f2553024e; no voters:
I20260926 03:03:05.288738 16376 raft_consensus.cc:2781] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:03:05.288880 16376 raft_consensus.cc:482] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:03:05.289031 16376 raft_consensus.cc:3037] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:03:05.291200 16376 raft_consensus.cc:504] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.291834 16376 leader_election.cc:288] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 election: Requested vote from peers 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063), fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445)
I20260926 03:03:05.292152 16174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9913752e47fd4df6a1337903aada59f5" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec"
I20260926 03:03:05.292236 16040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9913752e47fd4df6a1337903aada59f5" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3741b86c3b4020a6265d8f2553024e"
I20260926 03:03:05.292331 16174 raft_consensus.cc:3037] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:03:05.292470 16040 raft_consensus.cc:3037] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:03:05.294589 16174 raft_consensus.cc:2454] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 1.
I20260926 03:03:05.294598 16040 raft_consensus.cc:2454] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 1.
I20260926 03:03:05.294974 16242 leader_election.cc:302] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133119c685e47ed82ac25004c5d4265, 916eb5d32cbb4c9a9a3e233d4fabcfec; no voters:
I20260926 03:03:05.295320 16376 raft_consensus.cc:2781] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:03:05.295758 16376 raft_consensus.cc:686] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 1 LEADER]: Becoming Leader. State: Replica: 7133119c685e47ed82ac25004c5d4265, State: Running, Role: LEADER
I20260926 03:03:05.296202 16376 consensus_queue.cc:237] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } }
I20260926 03:03:05.300422 15896 catalog_manager.cc:5495] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133119c685e47ed82ac25004c5d4265 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7133119c685e47ed82ac25004c5d4265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } health_report { overall_health: UNKNOWN } } }
I20260926 03:03:05.315428 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.310428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.0.97.3:40987)
0926 03:03:05.310583 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.310587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.310670 (+ 83us) server_negotiation.cc:407] Connection header received
0926 03:03:05.310865 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.310868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.310938 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.311052 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:03:05.311367 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.312162 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.313621 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.313997 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.314085 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.314515 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:03:05.314966 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:03:05.315066 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.315160 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.315218 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:03:05.322751 16040 raft_consensus.cc:1261] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Refusing update from remote peer 7133119c685e47ed82ac25004c5d4265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:03:05.322751 16174 raft_consensus.cc:1261] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Refusing update from remote peer 7133119c685e47ed82ac25004c5d4265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:03:05.323647 16380 consensus_queue.cc:1035] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:03:05.323963 16376 consensus_queue.cc:1035] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:03:05.334554 16383 mvcc.cc:205] Tried to move back new op lower bound from 7333444752672960512 to 7333444752577187840. Current Snapshot: MvccSnapshot[applied={T|T < 7333444752672960512}]
I20260926 03:03:05.348513 15895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:03:05.348805 15895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:03:05.352557 15895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Sending DeleteTablet for 3 replicas of tablet 9913752e47fd4df6a1337903aada59f5
I20260926 03:03:05.353739 16287 tablet_service.cc:1514] Processing DeleteTablet for tablet 9913752e47fd4df6a1337903aada59f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:03:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48122
I20260926 03:03:05.353751 16020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9913752e47fd4df6a1337903aada59f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:03:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58080
I20260926 03:03:05.353895 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 15957
I20260926 03:03:05.354614 16154 tablet_service.cc:1514] Processing DeleteTablet for tablet 9913752e47fd4df6a1337903aada59f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:03:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50584
I20260926 03:03:05.366300 16394 tablet_replica.cc:330] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: stopping tablet replica
I20260926 03:03:05.366990 16394 raft_consensus.cc:2229] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:03:05.367614 16394 raft_consensus.cc:2258] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:03:05.370127 16394 ts_tablet_manager.cc:1905] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:03:05.377367 16394 ts_tablet_manager.cc:1918] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:03:05.377605 16394 log.cc:1196] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/9913752e47fd4df6a1337903aada59f5
Sep 26 03:03:05 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:03:05.378283 16394 ts_tablet_manager.cc:1939] T 9913752e47fd4df6a1337903aada59f5 P fb3741b86c3b4020a6265d8f2553024e: Deleting consensus metadata
I20260926 03:03:05.380229 15881 catalog_manager.cc:4841] TS fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445): tablet 9913752e47fd4df6a1337903aada59f5 (table test-table [id=1be46c8552224196a15bbc861870f223]) successfully deleted
I20260926 03:03:05.381309 16396 tablet_replica.cc:330] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: stopping tablet replica
I20260926 03:03:05.381839 16396 raft_consensus.cc:2229] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:03:05.382476 16396 raft_consensus.cc:2258] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:03:05.383598 16393 tablet_replica.cc:330] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: stopping tablet replica
I20260926 03:03:05.384043 16393 raft_consensus.cc:2229] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:03:05.384732 16393 raft_consensus.cc:2258] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:03:05.385242 16396 ts_tablet_manager.cc:1905] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:03:05.386590 16393 ts_tablet_manager.cc:1905] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:03:05.388005 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.355187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.0.97.1:41445)
0926 03:03:05.355350 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.355356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.367147 (+ 11791us) server_negotiation.cc:407] Connection header received
0926 03:03:05.367438 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.367446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.367526 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.367620 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:05.369881 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.370898 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.372809 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.373439 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.373535 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.379228 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:05.379238 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:05.379256 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:05.379296 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:05.383693 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.384671 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.384680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.384684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.384888 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.385417 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.385422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.385425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.385950 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:05.386233 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.387421 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.387637 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:03:05.388796 16080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:05.392045 16396 ts_tablet_manager.cc:1918] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:03:05.392194 16396 log.cc:1196] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/9913752e47fd4df6a1337903aada59f5
I20260926 03:03:05.392709 16396 ts_tablet_manager.cc:1939] T 9913752e47fd4df6a1337903aada59f5 P 916eb5d32cbb4c9a9a3e233d4fabcfec: Deleting consensus metadata
I20260926 03:03:05.394480 16393 ts_tablet_manager.cc:1918] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:03:05.394433 15882 catalog_manager.cc:4841] TS 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063): tablet 9913752e47fd4df6a1337903aada59f5 (table test-table [id=1be46c8552224196a15bbc861870f223]) successfully deleted
I20260926 03:03:05.394693 16393 log.cc:1196] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/9913752e47fd4df6a1337903aada59f5
I20260926 03:03:05.395308 16393 ts_tablet_manager.cc:1939] T 9913752e47fd4df6a1337903aada59f5 P 7133119c685e47ed82ac25004c5d4265: Deleting consensus metadata
I20260926 03:03:05.397168 15882 catalog_manager.cc:4841] TS 7133119c685e47ed82ac25004c5d4265 (127.0.97.3:40987): tablet 9913752e47fd4df6a1337903aada59f5 (table test-table [id=1be46c8552224196a15bbc861870f223]) successfully deleted
I20260926 03:03:05.484110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 15957
I20260926 03:03:05.501581 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 16091
Sep 26 03:03:05 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:03:05.517613 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.502409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.0.97.2:43063)
0926 03:03:05.502580 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.502587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.502627 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:03:05.502846 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.502851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.502923 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.503037 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:05.504357 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.505169 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.506522 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.506946 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.507080 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.511927 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:05.511949 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:05.511965 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:05.512004 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:05.514906 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.515721 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.515728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.515730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.515844 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.516247 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.516250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.516252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.516605 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:05.516815 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.517179 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.517333 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:03:05.518297 16214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:05.675014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 16091
I20260926 03:03:05.692324 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 16224
Sep 26 03:03:05 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:03:05.709390 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:05.693266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.0.97.3:40987)
0926 03:03:05.693472 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:05.693481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:05.693507 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:03:05.693574 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:05.693580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:05.693663 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:05.693790 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:05.695190 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.695999 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.697500 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:05.697959 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:05.698106 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:05.702535 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:05.702553 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:05.702573 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:05.702618 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:05.706497 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.707170 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.707181 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.707187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.707379 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:05.707738 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:05.707743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:05.707746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:05.708208 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:05.708541 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:05.708720 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:03:05.708945 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:03:05.710301 16347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:05.832554 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 16224
I20260926 03:03:05.853860 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 15864
I20260926 03:03:05.855003 15926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:05.962881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 15864
I20260926 03:03:25.984478 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37913
--webserver_interface=127.0.97.62
--webserver_port=44347
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:03:26.319708 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:26.319984 16401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:26.320135 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:26.327416 16401 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:03:26.327510 16401 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:03:26.327574 16401 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:03:26.327654 16401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:03:26.327750 16401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:26.327860 16401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:26.327947 16401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:03:26.328028 16401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:03:26.337924 16401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:26.339179 16401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:26.340508 16401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:26.363533 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:26.363909 16401 server_base.cc:1034] running on GCE node
W20260926 03:03:26.363524 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:26.363565 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:26.365173 16401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:26.366828 16401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:26.368089 16401 hybrid_clock.cc:648] HybridClock initialized: now 1790391806368016 us; error 47 us; skew 500 ppm
I20260926 03:03:26.368597 16401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:26 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:26.372427 16401 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:03:26.374117 16401 webserver.cc:460] Webserver started at http://127.0.97.62:44347/ using document root <none> and password file <none>
I20260926 03:03:26.374938 16401 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:26.375099 16401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:26.381208 16401 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 03:03:26.384058 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:26.385492 16401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:03:26.385677 16401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5"
format_stamp: "Formatted at 2026-09-26 03:03:03 on dist-test-slave-fccw"
server_key: "87d1b85414785bf33ecdf6c07aee4494"
server_key_iv: "f32f48af6ef3af7af3618cdca23557e6"
server_key_version: "encryptionkey@0"
I20260926 03:03:26.386796 16401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:26.405340 16401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:26.409488 16401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:26.409785 16401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:26.428674 16401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37913
I20260926 03:03:26.428697 16468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37913 every 8 connection(s)
I20260926 03:03:26.430263 16401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:03:26.437381 16469 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:03:26.437572 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16401
I20260926 03:03:26.438359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41445
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37913
--webserver_port=41815
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:03:26.440727 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Bootstrap starting.
I20260926 03:03:26.467401 16469 log.cc:824] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:26.482561 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:03:26.483211 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Bootstrap complete.
I20260926 03:03:26.496752 16469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:26.498154 16469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5244f0a890cb452ea1e2ca44ef3d3eb5, State: Initialized, Role: FOLLOWER
I20260926 03:03:26.498852 16469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:26.499130 16469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:03:26.499255 16469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:03:26.499436 16469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:03:26.505689 16469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:26.506412 16469 leader_election.cc:302] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5244f0a890cb452ea1e2ca44ef3d3eb5; no voters:
I20260926 03:03:26.507145 16469 leader_election.cc:288] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:03:26.507302 16473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:03:26.508713 16473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 LEADER]: Becoming Leader. State: Replica: 5244f0a890cb452ea1e2ca44ef3d3eb5, State: Running, Role: LEADER
I20260926 03:03:26.509433 16469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:03:26.509330 16473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:26.511618 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5244f0a890cb452ea1e2ca44ef3d3eb5. Latest consensus state: current_term: 2 leader_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } } }
I20260926 03:03:26.512048 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:03:26.512629 16474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } } }
I20260926 03:03:26.513034 16474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:03:26.522140 16479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:03:26.531553 16479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:03:26.533025 16479 catalog_manager.cc:1259] Loaded cluster ID: a65fcab1744e41e79a544b3965e06545
I20260926 03:03:26.533157 16479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:03:26.537259 16479 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:03:26.539006 16479 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Loaded TSK: 0
I20260926 03:03:26.585256 16491 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:03:26.587594 16479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Generated new TSK 1
I20260926 03:03:26.588004 16479 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:03:26.823676 16471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:26.824015 16471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:26.824183 16471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:26.831962 16471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:26.832065 16471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:26.832238 16471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:03:26.842588 16471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41815
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:26.844041 16471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:26.845479 16471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:26.859751 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:26.865033 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:26.865952 16471 server_base.cc:1034] running on GCE node
W20260926 03:03:26.865996 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:26.866706 16471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:26.867758 16471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:26.869186 16471 hybrid_clock.cc:648] HybridClock initialized: now 1790391806868922 us; error 66 us; skew 500 ppm
I20260926 03:03:26.869545 16471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:26 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:26.872750 16471 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:03:26.874415 16471 webserver.cc:460] Webserver started at http://127.0.97.1:41815/ using document root <none> and password file <none>
I20260926 03:03:26.875099 16471 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:26.875254 16471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:26.880761 16471 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.002s
I20260926 03:03:26.883697 16507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:26.885272 16471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:03:26.885479 16471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "fb3741b86c3b4020a6265d8f2553024e"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "5e50ac8254bdcaba279093e43db605a8"
server_key_iv: "a61afcbb37d1db970eb5eb28093af889"
server_key_version: "encryptionkey@0"
I20260926 03:03:26.886446 16471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:26.944032 16471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:26.948522 16471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:26.948805 16471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:26.949838 16471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:26.951131 16471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:26.951246 16471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:26.951376 16471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:26.951469 16471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:26.984421 16471 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41445
I20260926 03:03:26.984514 16619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41445 every 8 connection(s)
I20260926 03:03:26.986006 16471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:03:26.996435 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16471
I20260926 03:03:26.997164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43063
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37913
--webserver_port=33793
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:03:26 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:03:27.010578 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:26.988840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44319 (local address 127.0.97.62:37913)
0926 03:03:26.989507 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:26.989525 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:26.990409 (+ 884us) server_negotiation.cc:407] Connection header received
0926 03:03:26.992926 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:26.992939 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:26.993378 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:26.993753 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:26.995360 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:26.996677 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:26.998402 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:26.999045 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:26.999258 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.002883 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:27.002896 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:27.002918 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:27.002970 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:27.005961 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.006916 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.006923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.006930 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.007078 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.007580 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.007583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.007585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.008039 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:27.008183 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:27.009146 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 03:03:27.009507 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
I20260926 03:03:27.010931 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:26.989195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:44319)
0926 03:03:26.990223 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:26.990272 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:26.992359 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:26.993907 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:26.993918 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:26.994592 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:03:26.995143 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:26.995161 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:26.996844 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:26.996849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:26.998244 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:26.998257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:26.999291 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:26.999308 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:26.999587 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.000284 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:03:27.000317 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:03:27.002598 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:03:27.006106 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:27.006118 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:27.006137 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:27.006771 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:27.007190 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:27.007208 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:27.007215 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:27.007468 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:27.008181 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:03:27.008191 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:03:27.008613 (+ 422us) client_negotiation.cc:769] Sending connection context
0926 03:03:27.008926 (+ 313us) client_negotiation.cc:240] Negotiation successful
0926 03:03:27.009578 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":389,"threads_started":1}
I20260926 03:03:27.014001 16620 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37913
I20260926 03:03:27.014331 16620 heartbeater.cc:464] Registering TS with master...
I20260926 03:03:27.015210 16620 heartbeater.cc:511] Master 127.0.97.62:37913 requested a full tablet report, sending...
I20260926 03:03:27.017894 16434 ts_manager.cc:194] Registered new tserver with Master: fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445)
I20260926 03:03:27.020236 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44319
W20260926 03:03:27.328359 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:27.328649 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:27.328743 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:27.335997 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:27.336112 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:27.336234 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:03:27.346035 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33793
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:27.347294 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:27.348577 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:27.363896 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:27.363874 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:27.364853 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:27.365124 16624 server_base.cc:1034] running on GCE node
I20260926 03:03:27.365787 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:27.366740 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:27.367969 16624 hybrid_clock.cc:648] HybridClock initialized: now 1790391807367913 us; error 51 us; skew 500 ppm
I20260926 03:03:27.368336 16624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:27 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:27.376047 16624 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:03:27.377897 16624 webserver.cc:460] Webserver started at http://127.0.97.2:33793/ using document root <none> and password file <none>
I20260926 03:03:27.378775 16624 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:27.378942 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:27.385181 16624 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:03:27.388090 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:27.389616 16624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:03:27.389797 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "98a2feaa704638c9d907875b0e4b16ba"
server_key_iv: "311eacb1d951af0da8dd3bc0034be445"
server_key_version: "encryptionkey@0"
I20260926 03:03:27.390911 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:27.428411 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:27.432490 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:27.432756 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:27.433877 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:27.435648 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:27.435763 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:27.435889 16624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:27.435983 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:27.471316 16624 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43063
I20260926 03:03:27.471374 16751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43063 every 8 connection(s)
I20260926 03:03:27.472757 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:03:27.473240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16624
I20260926 03:03:27.474025 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40987
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37913
--webserver_port=40975
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:03:27 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:03:27.496572 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.475522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55377 (local address 127.0.97.62:37913)
0926 03:03:27.475753 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:27.475762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:27.477031 (+ 1269us) server_negotiation.cc:407] Connection header received
0926 03:03:27.479916 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:27.479921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:27.480006 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:27.480097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:27.482227 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.483406 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.484898 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.485681 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.485823 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.489473 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:27.489482 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:27.489490 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:27.489535 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:27.493257 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.494127 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.494135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.494141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.494326 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.494793 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.494797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.494799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.495067 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:27.495228 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:27.496000 (+ 772us) server_negotiation.cc:299] Negotiation successful
0926 03:03:27.496223 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:03:27.497601 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.476006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:55377)
0926 03:03:27.476822 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:27.476879 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:27.479606 (+ 2727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:27.480378 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:27.480393 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:27.481205 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:03:27.482039 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:27.482060 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.483558 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.483562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:27.484747 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:27.484758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.485884 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.485889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:27.486078 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.486755 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:03:27.486775 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:03:27.489235 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:03:27.493523 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:27.493529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:27.493544 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:27.493962 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:27.494469 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:27.494474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:27.494481 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:27.494663 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:27.495220 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:03:27.495227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:03:27.495637 (+ 410us) client_negotiation.cc:769] Sending connection context
0926 03:03:27.495900 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 03:03:27.496450 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":287,"threads_started":1}
I20260926 03:03:27.499205 16752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37913
I20260926 03:03:27.499505 16752 heartbeater.cc:464] Registering TS with master...
I20260926 03:03:27.500198 16752 heartbeater.cc:511] Master 127.0.97.62:37913 requested a full tablet report, sending...
I20260926 03:03:27.501638 16434 ts_manager.cc:194] Registered new tserver with Master: 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063)
I20260926 03:03:27.502733 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55377
W20260926 03:03:27.804143 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:27.804407 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:27.804512 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:27.811599 16755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:27.811692 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:27.811811 16755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:03:27.820896 16755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40975
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:27.822144 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:27.823351 16755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:27.835907 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:27.836071 16761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:27.836045 16763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:27.837147 16755 server_base.cc:1034] running on GCE node
I20260926 03:03:27.837667 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:27.838490 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:27.839675 16755 hybrid_clock.cc:648] HybridClock initialized: now 1790391807839637 us; error 33 us; skew 500 ppm
I20260926 03:03:27.839977 16755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:27 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:27.842974 16755 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:03:27.844359 16755 webserver.cc:460] Webserver started at http://127.0.97.3:40975/ using document root <none> and password file <none>
I20260926 03:03:27.845081 16755 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:27.845237 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:27.850359 16755 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:03:27.853144 16770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:27.854583 16755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:03:27.854749 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "7133119c685e47ed82ac25004c5d4265"
format_stamp: "Formatted at 2026-09-26 03:03:05 on dist-test-slave-fccw"
server_key: "2c38527667367f9eb2941a56eb961563"
server_key_iv: "0f37e7a9ff6f74c5f1e12e04d287c34f"
server_key_version: "encryptionkey@0"
I20260926 03:03:27.855702 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:27.875672 16755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:27.879596 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:27.879858 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:27.880818 16755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:27.882104 16755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:27.882231 16755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:27.882368 16755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:27.882458 16755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:27.915320 16755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40987
I20260926 03:03:27.915407 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40987 every 8 connection(s)
I20260926 03:03:27.916765 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:03:27.920569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16755
Sep 26 03:03:27 dist-test-slave-fccw krb5kdc[15851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:03:27.936875 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.921592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.0.97.62:37913)
0926 03:03:27.922338 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:27.922342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:27.922358 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:03:27.922409 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:27.922413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:27.922468 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:27.922557 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:27.924634 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.925441 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.926798 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.927211 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.927369 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.930333 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:27.930338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:27.930341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:27.930371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:27.934135 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.934771 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.934777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.934780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.934940 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.935352 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.935356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.935359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.935657 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:27.935909 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:27.936250 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:03:27.936467 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":550,"threads_started":1}
I20260926 03:03:27.938010 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.919299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39219 (local address 127.0.97.62:37913)
0926 03:03:27.919478 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:27.919482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:27.920579 (+ 1097us) server_negotiation.cc:407] Connection header received
0926 03:03:27.922947 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:27.922950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:27.922993 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:27.923060 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:27.924978 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.925775 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.927259 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.927705 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.927828 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.931390 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:27.931401 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:27.931406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:27.931458 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:27.934792 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.935598 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.935602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.935605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.935714 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.936166 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.936171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.936174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.936523 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:27.936659 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:27.937466 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 03:03:27.937674 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":9088}
I20260926 03:03:27.939175 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.919742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:39219)
0926 03:03:27.920419 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:27.920456 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:27.922654 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:27.923352 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:27.923365 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:27.924113 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:03:27.924807 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:27.924825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.925940 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.925945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:27.927104 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:27.927115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.927947 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.927954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:27.928169 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.928748 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:03:27.928772 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:03:27.931170 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:03:27.934910 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:27.934915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:27.934927 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:27.935494 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:27.935827 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:03:27.935832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:03:27.935835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:03:27.936063 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:03:27.936675 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:03:27.936683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:03:27.937040 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 03:03:27.937380 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 03:03:27.937882 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":291,"threads_started":1}
I20260926 03:03:27.940392 16883 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37913
I20260926 03:03:27.940675 16883 heartbeater.cc:464] Registering TS with master...
I20260926 03:03:27.941439 16883 heartbeater.cc:511] Master 127.0.97.62:37913 requested a full tablet report, sending...
I20260926 03:03:27.942690 16434 ts_manager.cc:194] Registered new tserver with Master: 7133119c685e47ed82ac25004c5d4265 (127.0.97.3:40987)
I20260926 03:03:27.943430 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39219
I20260926 03:03:27.951525 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:03:27.957652 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.952339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.0.97.62:37913)
0926 03:03:27.952542 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:27.952551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:27.952917 (+ 366us) server_negotiation.cc:407] Connection header received
0926 03:03:27.953125 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:27.953129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:27.953255 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:27.953399 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:03:27.953810 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.954557 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.955948 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.956354 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.956446 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.956865 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:03:27.957145 (+ 280us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 03:03:27.957417 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38028: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 03:03:27.957772 16622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38028: token expired
W20260926 03:03:27.958372 16355 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:37913)
I20260926 03:03:27.970412 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.958758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.0.97.62:37913)
0926 03:03:27.958962 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:27.958966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:27.958981 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:03:27.959163 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:27.959166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:27.959217 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:27.959342 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:27.960702 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.961458 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.962702 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:27.963070 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:27.963156 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:27.965651 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:27.965656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:27.965659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:27.965687 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:27.968016 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.968598 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.968601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.968603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.968757 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:27.969142 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:27.969147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:27.969150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:27.969439 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:27.969635 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:27.969917 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:03:27.970062 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:03:27.973671 16434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:03:27.980718 16434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:03:28.017979 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.993407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.0.97.1:41445)
0926 03:03:28.009963 (+ 16556us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.009972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.009998 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:03:28.010089 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.010097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.010389 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.010618 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:28.011358 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.013090 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.015785 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.016748 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.016923 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.017192 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.017447 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.017613 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16392,"thread_start_us":16221,"threads_started":1}
I20260926 03:03:28.018574 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.992845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41445 (local address 127.0.0.1:60190)
0926 03:03:27.993994 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:27.994029 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:27.994193 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:28.010590 (+ 16397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:28.010595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:28.010623 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:28.011125 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.011138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.013218 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.013226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.015649 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.015660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.016987 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.017021 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.017207 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.017260 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 03:03:28.017405 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 03:03:28.017592 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":67,"thread_start_us":14963,"threads_started":1}
I20260926 03:03:28.018769 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.993974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.0.97.2:43063)
0926 03:03:28.009416 (+ 15442us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.009429 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.009472 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:03:28.009595 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.009605 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.009893 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.010105 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:28.010900 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.012345 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.015821 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.017324 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.017436 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.017625 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.018278 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.018457 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15274,"thread_start_us":15174,"threads_started":1}
I20260926 03:03:28.019300 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.993136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43063 (local address 127.0.0.1:48160)
0926 03:03:28.007535 (+ 14399us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:28.007559 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:28.007710 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:28.010106 (+ 2396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:28.010113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:28.010149 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:28.010602 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.010614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.012494 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.012500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.015644 (+ 3144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.015657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.017839 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.017846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.018085 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.018142 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 03:03:28.018237 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:03:28.018351 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14228,"mutex_wait_us":78,"thread_start_us":482,"threads_started":1}
I20260926 03:03:28.019508 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.994886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.0.97.3:40987)
0926 03:03:28.009565 (+ 14679us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.009576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.009612 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:03:28.009716 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.009724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.009967 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.010157 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:28.011246 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.012899 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.015821 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.017314 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.017436 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.017657 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.019007 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.019184 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14502,"thread_start_us":14334,"threads_started":1}
I20260926 03:03:28.020013 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:27.994397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40987 (local address 127.0.0.1:45746)
0926 03:03:28.008432 (+ 14035us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:28.008455 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:28.008597 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:28.010452 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:28.010457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:28.010490 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:28.011025 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.011038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.013096 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.013101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.015644 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.015658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.017858 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.017864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.018611 (+ 747us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.018641 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:03:28.018729 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:03:28.018868 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13842,"mutex_wait_us":113,"spinlock_wait_cycles":1408,"thread_start_us":14304,"threads_started":1}
I20260926 03:03:28.022985 16818 tablet_service.cc:1467] Processing CreateTablet for tablet efded1faa38a48e49a0125aa5c81126b (DEFAULT_TABLE table=test-table [id=39e7b23b960349e49b78902bccf1ddfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:03:28.023022 16555 tablet_service.cc:1467] Processing CreateTablet for tablet efded1faa38a48e49a0125aa5c81126b (DEFAULT_TABLE table=test-table [id=39e7b23b960349e49b78902bccf1ddfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:03:28.023022 16687 tablet_service.cc:1467] Processing CreateTablet for tablet efded1faa38a48e49a0125aa5c81126b (DEFAULT_TABLE table=test-table [id=39e7b23b960349e49b78902bccf1ddfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:03:28.024410 16687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efded1faa38a48e49a0125aa5c81126b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:03:28.024413 16818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efded1faa38a48e49a0125aa5c81126b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:03:28.024411 16620 heartbeater.cc:503] Master 127.0.97.62:37913 was elected leader, sending a full tablet report...
I20260926 03:03:28.025143 16555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efded1faa38a48e49a0125aa5c81126b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:03:28.037988 16899 tablet_bootstrap.cc:492] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Bootstrap starting.
I20260926 03:03:28.039604 16900 tablet_bootstrap.cc:492] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Bootstrap starting.
I20260926 03:03:28.041473 16899 tablet_bootstrap.cc:654] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Neither blocks nor log segments found. Creating new log.
I20260926 03:03:28.041669 16901 tablet_bootstrap.cc:492] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Bootstrap starting.
I20260926 03:03:28.042783 16899 log.cc:824] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:28.043000 16900 tablet_bootstrap.cc:654] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Neither blocks nor log segments found. Creating new log.
I20260926 03:03:28.044088 16900 log.cc:824] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:28.044927 16901 tablet_bootstrap.cc:654] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Neither blocks nor log segments found. Creating new log.
I20260926 03:03:28.046092 16901 log.cc:824] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:28.046502 16899 tablet_bootstrap.cc:492] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: No bootstrap required, opened a new log
I20260926 03:03:28.046772 16899 ts_tablet_manager.cc:1397] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 03:03:28.047178 16900 tablet_bootstrap.cc:492] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: No bootstrap required, opened a new log
I20260926 03:03:28.047487 16900 ts_tablet_manager.cc:1397] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 03:03:28.051072 16901 tablet_bootstrap.cc:492] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: No bootstrap required, opened a new log
I20260926 03:03:28.051333 16901 ts_tablet_manager.cc:1397] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260926 03:03:28.057663 16899 raft_consensus.cc:348] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.057888 16900 raft_consensus.cc:348] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.058135 16899 raft_consensus.cc:374] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:03:28.058233 16899 raft_consensus.cc:729] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133119c685e47ed82ac25004c5d4265, State: Initialized, Role: FOLLOWER
I20260926 03:03:28.058267 16900 raft_consensus.cc:374] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:03:28.058363 16900 raft_consensus.cc:729] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb3741b86c3b4020a6265d8f2553024e, State: Initialized, Role: FOLLOWER
I20260926 03:03:28.058949 16899 consensus_queue.cc:260] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.058979 16900 consensus_queue.cc:260] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.059171 16901 raft_consensus.cc:348] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.059621 16901 raft_consensus.cc:374] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:03:28.059741 16901 raft_consensus.cc:729] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916eb5d32cbb4c9a9a3e233d4fabcfec, State: Initialized, Role: FOLLOWER
I20260926 03:03:28.060451 16883 heartbeater.cc:503] Master 127.0.97.62:37913 was elected leader, sending a full tablet report...
I20260926 03:03:28.060432 16901 consensus_queue.cc:260] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.061554 16752 heartbeater.cc:503] Master 127.0.97.62:37913 was elected leader, sending a full tablet report...
I20260926 03:03:28.062155 16901 ts_tablet_manager.cc:1428] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260926 03:03:28.062762 16900 ts_tablet_manager.cc:1428] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260926 03:03:28.063410 16899 ts_tablet_manager.cc:1428] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260926 03:03:28.168447 16884 tablet.cc:2369] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:03:28.170485 16906 raft_consensus.cc:482] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:03:28.170701 16906 raft_consensus.cc:504] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.172385 16906 leader_election.cc:288] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445), 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063)
I20260926 03:03:28.178007 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.172781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50123 (local address 127.0.97.1:41445)
0926 03:03:28.173041 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.173048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.173143 (+ 95us) server_negotiation.cc:407] Connection header received
0926 03:03:28.173372 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.173377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.173434 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.173528 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:28.173940 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.174698 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.176262 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.177131 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.177276 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.177377 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.177643 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.177763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 03:03:28.178041 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.172643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41445 (local address 127.0.97.3:50123)
0926 03:03:28.173031 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:28.173053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:28.173185 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:28.173550 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:28.173555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:28.173582 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:28.173771 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.173780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.174835 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.174841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.176089 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.176100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.177282 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.177287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.177477 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.177501 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:03:28.177616 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 03:03:28.177755 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260926 03:03:28.178674 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.172761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43063 (local address 127.0.97.3:50093)
0926 03:03:28.173639 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:28.173653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:03:28.173811 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:03:28.174097 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:03:28.174102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:03:28.174123 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:03:28.174530 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.174539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.175485 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.175491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.176778 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:03:28.176790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.177976 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.177982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:03:28.178198 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.178231 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:03:28.178315 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:03:28.178428 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":576,"threads_started":1}
I20260926 03:03:28.178810 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.172871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50093 (local address 127.0.97.2:43063)
0926 03:03:28.173123 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.173132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.173741 (+ 609us) server_negotiation.cc:407] Connection header received
0926 03:03:28.173941 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.173945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.174000 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.174086 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:03:28.174673 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.175359 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.176929 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.177756 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.177921 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.178015 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.178407 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.178541 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:03:28.179179 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efded1faa38a48e49a0125aa5c81126b" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3741b86c3b4020a6265d8f2553024e" is_pre_election: true
I20260926 03:03:28.179699 16707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efded1faa38a48e49a0125aa5c81126b" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" is_pre_election: true
I20260926 03:03:28.179801 16575 raft_consensus.cc:2454] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 0.
I20260926 03:03:28.180390 16707 raft_consensus.cc:2454] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 0.
I20260926 03:03:28.180642 16772 leader_election.cc:302] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133119c685e47ed82ac25004c5d4265, fb3741b86c3b4020a6265d8f2553024e; no voters:
I20260926 03:03:28.181049 16906 raft_consensus.cc:2781] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:03:28.181193 16906 raft_consensus.cc:482] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:03:28.181303 16906 raft_consensus.cc:3037] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:03:28.183429 16906 raft_consensus.cc:504] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.184065 16906 leader_election.cc:288] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 election: Requested vote from peers fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445), 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063)
I20260926 03:03:28.184475 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efded1faa38a48e49a0125aa5c81126b" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb3741b86c3b4020a6265d8f2553024e"
I20260926 03:03:28.184520 16707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efded1faa38a48e49a0125aa5c81126b" candidate_uuid: "7133119c685e47ed82ac25004c5d4265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec"
I20260926 03:03:28.184669 16575 raft_consensus.cc:3037] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:03:28.184702 16707 raft_consensus.cc:3037] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:03:28.186811 16575 raft_consensus.cc:2454] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 1.
I20260926 03:03:28.186918 16707 raft_consensus.cc:2454] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133119c685e47ed82ac25004c5d4265 in term 1.
I20260926 03:03:28.187222 16772 leader_election.cc:302] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133119c685e47ed82ac25004c5d4265, fb3741b86c3b4020a6265d8f2553024e; no voters:
I20260926 03:03:28.187506 16906 raft_consensus.cc:2781] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:03:28.187929 16906 raft_consensus.cc:686] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 1 LEADER]: Becoming Leader. State: Replica: 7133119c685e47ed82ac25004c5d4265, State: Running, Role: LEADER
I20260926 03:03:28.188380 16906 consensus_queue.cc:237] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } }
I20260926 03:03:28.192924 16434 catalog_manager.cc:5495] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133119c685e47ed82ac25004c5d4265 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7133119c685e47ed82ac25004c5d4265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133119c685e47ed82ac25004c5d4265" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40987 } health_report { overall_health: HEALTHY } } }
I20260926 03:03:28.206972 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.202341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.0.97.3:40987)
0926 03:03:28.202506 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.202510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.202626 (+ 116us) server_negotiation.cc:407] Connection header received
0926 03:03:28.202784 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.202788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.202856 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.202968 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:03:28.203253 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.203910 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.205164 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.205533 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.205620 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.206054 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:03:28.206540 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:03:28.206649 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.206723 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.206780 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:03:28.214725 16707 raft_consensus.cc:1261] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Refusing update from remote peer 7133119c685e47ed82ac25004c5d4265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:03:28.214726 16575 raft_consensus.cc:1261] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Refusing update from remote peer 7133119c685e47ed82ac25004c5d4265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:03:28.215721 16909 consensus_queue.cc:1035] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:03:28.216060 16906 consensus_queue.cc:1035] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb3741b86c3b4020a6265d8f2553024e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:03:28.225029 16753 tablet.cc:2369] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:03:28.230150 16912 mvcc.cc:205] Tried to move back new op lower bound from 7333444846437969920 to 7333444846343368704. Current Snapshot: MvccSnapshot[applied={T|T < 7333444846437969920}]
I20260926 03:03:28.242679 16433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:03:28.243007 16433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:03:28.246735 16433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Sending DeleteTablet for 3 replicas of tablet efded1faa38a48e49a0125aa5c81126b
I20260926 03:03:28.247870 16555 tablet_service.cc:1514] Processing DeleteTablet for tablet efded1faa38a48e49a0125aa5c81126b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:03:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60190
I20260926 03:03:28.247947 16687 tablet_service.cc:1514] Processing DeleteTablet for tablet efded1faa38a48e49a0125aa5c81126b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:03:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48160
I20260926 03:03:28.248178 16818 tablet_service.cc:1514] Processing DeleteTablet for tablet efded1faa38a48e49a0125aa5c81126b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:03:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45746
I20260926 03:03:28.249522 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 16471
I20260926 03:03:28.258360 16923 tablet_replica.cc:330] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: stopping tablet replica
I20260926 03:03:28.258829 16923 raft_consensus.cc:2229] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:03:28.259699 16923 raft_consensus.cc:2258] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:03:28.260800 16921 tablet_replica.cc:330] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: stopping tablet replica
I20260926 03:03:28.261210 16921 raft_consensus.cc:2229] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:03:28.261678 16921 raft_consensus.cc:2258] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:03:28.263340 16923 ts_tablet_manager.cc:1905] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:03:28.264433 16922 tablet_replica.cc:330] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: stopping tablet replica
I20260926 03:03:28.264847 16922 raft_consensus.cc:2229] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:03:28.265344 16922 raft_consensus.cc:2258] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:03:28.265800 16921 ts_tablet_manager.cc:1905] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:03:28.268023 16922 ts_tablet_manager.cc:1905] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:03:28.272639 16923 ts_tablet_manager.cc:1918] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:03:28.272805 16923 log.cc:1196] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/efded1faa38a48e49a0125aa5c81126b
I20260926 03:03:28.272857 16921 ts_tablet_manager.cc:1918] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:03:28.273065 16921 log.cc:1196] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/efded1faa38a48e49a0125aa5c81126b
I20260926 03:03:28.273411 16923 ts_tablet_manager.cc:1939] T efded1faa38a48e49a0125aa5c81126b P 7133119c685e47ed82ac25004c5d4265: Deleting consensus metadata
I20260926 03:03:28.273607 16921 ts_tablet_manager.cc:1939] T efded1faa38a48e49a0125aa5c81126b P fb3741b86c3b4020a6265d8f2553024e: Deleting consensus metadata
I20260926 03:03:28.275048 16420 catalog_manager.cc:4841] TS 7133119c685e47ed82ac25004c5d4265 (127.0.97.3:40987): tablet efded1faa38a48e49a0125aa5c81126b (table test-table [id=39e7b23b960349e49b78902bccf1ddfa]) successfully deleted
I20260926 03:03:28.275310 16419 catalog_manager.cc:4841] TS fb3741b86c3b4020a6265d8f2553024e (127.0.97.1:41445): tablet efded1faa38a48e49a0125aa5c81126b (table test-table [id=39e7b23b960349e49b78902bccf1ddfa]) successfully deleted
I20260926 03:03:28.280355 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.251359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.0.97.1:41445)
0926 03:03:28.251580 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.251588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.251610 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:03:28.251679 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.251686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.251800 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.251918 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:28.254478 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.255941 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.259819 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.260800 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.260955 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.265959 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:28.265974 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:28.265991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:28.266037 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:28.276990 (+ 10953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:28.277970 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:28.277980 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:28.277986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:28.278155 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:28.278616 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:28.278623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:28.278627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:28.279057 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:28.279400 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.279715 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.279945 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:03:28.280797 16922 ts_tablet_manager.cc:1918] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:03:28.280979 16922 log.cc:1196] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/efded1faa38a48e49a0125aa5c81126b
I20260926 03:03:28.281104 16615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:28.281677 16922 ts_tablet_manager.cc:1939] T efded1faa38a48e49a0125aa5c81126b P 916eb5d32cbb4c9a9a3e233d4fabcfec: Deleting consensus metadata
I20260926 03:03:28.284205 16420 catalog_manager.cc:4841] TS 916eb5d32cbb4c9a9a3e233d4fabcfec (127.0.97.2:43063): tablet efded1faa38a48e49a0125aa5c81126b (table test-table [id=39e7b23b960349e49b78902bccf1ddfa]) successfully deleted
I20260926 03:03:28.372231 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 16471
I20260926 03:03:28.389582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 16624
I20260926 03:03:28.403530 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.390384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.0.97.2:43063)
0926 03:03:28.390575 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.390583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.390635 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:03:28.390872 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.390878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.390955 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.391061 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:28.392449 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.393263 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.394621 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.395029 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.395181 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.397768 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:28.397790 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:28.397813 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:28.397849 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:28.400597 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:28.401428 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:28.401439 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:28.401443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:28.401575 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:28.401956 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:28.401961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:28.401965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:28.402349 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:28.402660 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.402961 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.403177 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:03:28.404126 16747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:28.556763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 16624
I20260926 03:03:28.574200 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 16755
I20260926 03:03:28.588531 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:28.574983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.0.97.3:40987)
0926 03:03:28.575180 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:03:28.575187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:03:28.575264 (+ 77us) server_negotiation.cc:407] Connection header received
0926 03:03:28.575512 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:03:28.575517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:03:28.575593 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:03:28.575706 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:03:28.577033 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.577850 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.579147 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:03:28.579593 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:03:28.579734 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:03:28.582431 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:03:28.582456 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:03:28.582472 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:03:28.582514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:03:28.585528 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:28.586238 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:28.586245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:28.586248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:28.586415 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:03:28.586809 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:03:28.586818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:03:28.586821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:03:28.587122 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:03:28.587472 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 03:03:28.587921 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 03:03:28.588176 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:03:28.589223 16878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:28.713204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 16755
I20260926 03:03:28.733717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 16401
I20260926 03:03:28.734774 16464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:03:28.848007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 16401
I20260926 03:03:48.870065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37913
--webserver_interface=127.0.97.62
--webserver_port=44347
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:03:49.193399 16928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:49.193668 16928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:49.193849 16928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:49.200937 16928 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 03:03:49.201048 16928 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 03:03:49.201086 16928 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 03:03:49.201167 16928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:03:49.201249 16928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:49.201401 16928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:49.201484 16928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:03:49.201565 16928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:03:49.211329 16928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:49.212595 16928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:49.213934 16928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:49.227072 16933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:49.227128 16936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:49.227104 16934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:49.227324 16928 server_base.cc:1034] running on GCE node
I20260926 03:03:49.228229 16928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:49.229584 16928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:49.230804 16928 hybrid_clock.cc:648] HybridClock initialized: now 1790391829230746 us; error 44 us; skew 500 ppm
I20260926 03:03:49.231174 16928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:49 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:49.234349 16928 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:03:49.235671 16928 webserver.cc:460] Webserver started at http://127.0.97.62:44347/ using document root <none> and password file <none>
I20260926 03:03:49.236356 16928 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:49.236485 16928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:49.241762 16928 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:03:49.244496 16943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:49.245801 16928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:03:49.245985 16928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5"
format_stamp: "Formatted at 2026-09-26 03:03:03 on dist-test-slave-fccw"
server_key: "87d1b85414785bf33ecdf6c07aee4494"
server_key_iv: "f32f48af6ef3af7af3618cdca23557e6"
server_key_version: "encryptionkey@0"
I20260926 03:03:49.246935 16928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:49.268560 16928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:49.272914 16928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:49.273208 16928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:49.290582 16928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37913
I20260926 03:03:49.290620 16994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37913 every 8 connection(s)
I20260926 03:03:49.292081 16928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:03:49.298750 16995 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:03:49.301944 16995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Bootstrap starting.
I20260926 03:03:49.302631 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16928
I20260926 03:03:49.303659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41445
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37913
--webserver_port=41815
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:03:49.325598 16995 log.cc:824] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Log is configured to *not* fsync() on all Append() calls
I20260926 03:03:49.341121 16995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:03:49.341665 16995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Bootstrap complete.
I20260926 03:03:49.350764 16995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:49.351732 16995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5244f0a890cb452ea1e2ca44ef3d3eb5, State: Initialized, Role: FOLLOWER
I20260926 03:03:49.352312 16995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:49.352623 16995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:03:49.352744 16995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:03:49.352931 16995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 2 FOLLOWER]: Advancing to term 3
I20260926 03:03:49.356621 16995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:49.357159 16995 leader_election.cc:302] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5244f0a890cb452ea1e2ca44ef3d3eb5; no voters:
I20260926 03:03:49.357857 16995 leader_election.cc:288] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 03:03:49.357993 16999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 03:03:49.359241 16999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [term 3 LEADER]: Becoming Leader. State: Replica: 5244f0a890cb452ea1e2ca44ef3d3eb5, State: Running, Role: LEADER
I20260926 03:03:49.359782 16999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } }
I20260926 03:03:49.360093 16995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:03:49.362298 17001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5244f0a890cb452ea1e2ca44ef3d3eb5. Latest consensus state: current_term: 3 leader_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } } }
I20260926 03:03:49.362663 17001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:03:49.362406 17000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5244f0a890cb452ea1e2ca44ef3d3eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37913 } } }
I20260926 03:03:49.362839 17000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5 [sys.catalog]: This master's current role is: LEADER
I20260926 03:03:49.373966 17006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:03:49.382808 17006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:03:49.384207 17006 catalog_manager.cc:1259] Loaded cluster ID: a65fcab1744e41e79a544b3965e06545
I20260926 03:03:49.384316 17006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:03:49.388244 17006 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:03:49.389806 17006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Loaded TSK: 0
I20260926 03:03:49.389958 17006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5244f0a890cb452ea1e2ca44ef3d3eb5: Loaded TSK: 1
F20260926 03:03:49.391062 17006 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f35c2c3b9ad google::LogMessage::Fail() at ??:0
@ 0x7f35c2c3e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f35c2c3b4de google::LogMessage::Flush() at ??:0
@ 0x7f35c2c3ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f35c699f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f35c698e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f35d866a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f35d8669783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f35d86f52a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f35d86f50c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f35d873f5e9 std::function<>::operator()() at ??:0
@ 0x7f35d866dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f35d866bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f35d86f3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f35d86f3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f35c5887345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f35c588a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f35c588a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f35c5865eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f35c8b3f6db start_thread at ??:0
@ 0x7f35c1ad561f clone at ??:0
W20260926 03:03:49.698426 16997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:49.698691 16997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:49.698784 16997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:49.706099 16997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:49.706197 16997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:49.706398 16997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:03:49.715848 16997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41815
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:49.717152 16997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:49.718441 16997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:49.731406 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:49.731531 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:49.731503 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:49.732380 16997 server_base.cc:1034] running on GCE node
I20260926 03:03:49.733120 16997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:49.734035 16997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:49.735255 16997 hybrid_clock.cc:648] HybridClock initialized: now 1790391829735214 us; error 56 us; skew 500 ppm
I20260926 03:03:49.735572 16997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:49 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:49.739154 16997 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:03:49.740589 16997 webserver.cc:460] Webserver started at http://127.0.97.1:41815/ using document root <none> and password file <none>
I20260926 03:03:49.741331 16997 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:49.741468 16997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:49.746724 16997 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 03:03:49.749439 17052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:49.750743 16997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:03:49.750931 16997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "fb3741b86c3b4020a6265d8f2553024e"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "5e50ac8254bdcaba279093e43db605a8"
server_key_iv: "a61afcbb37d1db970eb5eb28093af889"
server_key_version: "encryptionkey@0"
I20260926 03:03:49.751917 16997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:49.779363 16997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:49.783474 16997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:49.783758 16997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:49.784740 16997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:49.786125 16997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:49.786257 16997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:49.786384 16997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:49.786473 16997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:49.820966 16997 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41445
I20260926 03:03:49.821039 17164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41445 every 8 connection(s)
I20260926 03:03:49.822422 16997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:03:49.827507 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:49.825552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:33799)
0926 03:03:49.826335 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:49.826543 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":306,"threads_started":1}
W20260926 03:03:49.828127 17165 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37913 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37913: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
I20260926 03:03:49.828927 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:49.828619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:45247)
0926 03:03:49.828763 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:49.828833 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 03:03:49.829759 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:49.829431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:55141)
0926 03:03:49.829581 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:49.829659 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 03:03:49.829996 17165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:03:49.831416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 16997
I20260926 03:03:49.832093 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43063
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37913
--webserver_port=33793
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:03:50.152164 17168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:50.152442 17168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:50.152532 17168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:50.159592 17168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:50.159696 17168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:50.159809 17168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:03:50.169202 17168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33793
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:50.170457 17168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:50.171708 17168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:50.184613 17173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:50.184700 17174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:50.184657 17176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:50.185446 17168 server_base.cc:1034] running on GCE node
I20260926 03:03:50.186049 17168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:50.186944 17168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:50.188165 17168 hybrid_clock.cc:648] HybridClock initialized: now 1790391830188101 us; error 58 us; skew 500 ppm
I20260926 03:03:50.188488 17168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:50 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:50.191480 17168 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:03:50.192785 17168 webserver.cc:460] Webserver started at http://127.0.97.2:33793/ using document root <none> and password file <none>
I20260926 03:03:50.193481 17168 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:50.193615 17168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:50.198763 17168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 03:03:50.201412 17183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:50.202618 17168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:03:50.202780 17168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "916eb5d32cbb4c9a9a3e233d4fabcfec"
format_stamp: "Formatted at 2026-09-26 03:03:04 on dist-test-slave-fccw"
server_key: "98a2feaa704638c9d907875b0e4b16ba"
server_key_iv: "311eacb1d951af0da8dd3bc0034be445"
server_key_version: "encryptionkey@0"
I20260926 03:03:50.203734 17168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:50.220021 17168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:50.223706 17168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:50.223961 17168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:50.224985 17168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:50.226311 17168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:50.226434 17168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:50.226562 17168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:50.226647 17168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:50.259388 17168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43063
I20260926 03:03:50.259440 17295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43063 every 8 connection(s)
I20260926 03:03:50.260811 17168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:03:50.265327 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:50.263592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:44029)
0926 03:03:50.264276 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:50.264459 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":276,"threads_started":1}
I20260926 03:03:50.265496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 17168
W20260926 03:03:50.266093 17296 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37913 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37913: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
I20260926 03:03:50.266295 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40987
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37913
--webserver_port=40975
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:03:50.266970 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:50.266671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:37205)
0926 03:03:50.266809 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:50.266880 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:03:50.267868 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:50.267579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:37153)
0926 03:03:50.267697 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:50.267762 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:03:50.268149 17296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:03:50.581967 17299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:03:50.582212 17299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:03:50.582412 17299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:03:50.589408 17299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:03:50.589541 17299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:03:50.589731 17299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:03:50.598927 17299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40975
--tserver_master_addrs=127.0.97.62:37913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:03:50.600209 17299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:03:50.601517 17299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:03:50.614245 17304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:50.614276 17305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:03:50.614336 17307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:03:50.615190 17299 server_base.cc:1034] running on GCE node
I20260926 03:03:50.615729 17299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:03:50.616580 17299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:03:50.617765 17299 hybrid_clock.cc:648] HybridClock initialized: now 1790391830617706 us; error 59 us; skew 500 ppm
I20260926 03:03:50.618081 17299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:03:50 dist-test-slave-fccw krb5kdc[15851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:03:50.620949 17299 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:03:50.622471 17299 webserver.cc:460] Webserver started at http://127.0.97.3:40975/ using document root <none> and password file <none>
I20260926 03:03:50.623137 17299 fs_manager.cc:359] Metadata directory not provided
I20260926 03:03:50.623277 17299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:03:50.628350 17299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:03:50.630914 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:50.632195 17299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:03:50.632361 17299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "7133119c685e47ed82ac25004c5d4265"
format_stamp: "Formatted at 2026-09-26 03:03:05 on dist-test-slave-fccw"
server_key: "2c38527667367f9eb2941a56eb961563"
server_key_iv: "0f37e7a9ff6f74c5f1e12e04d287c34f"
server_key_version: "encryptionkey@0"
I20260926 03:03:50.633358 17299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:03:50.668677 17299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:03:50.672612 17299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:03:50.672899 17299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:03:50.673879 17299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:03:50.675176 17299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:03:50.675304 17299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:50.675432 17299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:03:50.675518 17299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:03:50.708173 17299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40987
I20260926 03:03:50.708215 17426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40987 every 8 connection(s)
I20260926 03:03:50.709672 17299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:03:50.710704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 17299
I20260926 03:03:50.714551 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:50.712814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:34315)
0926 03:03:50.713541 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:50.713743 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":305,"threads_started":1}
W20260926 03:03:50.715287 17427 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37913 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37913: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
I20260926 03:03:50.715973 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:50.715703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:33447)
0926 03:03:50.715820 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:50.715881 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 03:03:50.716738 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:50.716441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:45401)
0926 03:03:50.716570 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:50.716636 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 03:03:50.717036 17427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 03:03:50.831251 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:50.830506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:44785)
0926 03:03:50.830947 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:50.831027 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":219,"threads_started":1}
I20260926 03:03:51.269424 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:51.268622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:44919)
0926 03:03:51.269049 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:51.269191 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":202,"threads_started":1}
I20260926 03:03:51.718454 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:51.717544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:54951)
0926 03:03:51.718018 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:51.718153 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260926 03:03:51.832690 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:51.832022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:55401)
0926 03:03:51.832459 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:51.832531 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":218,"threads_started":1}
I20260926 03:03:52.271000 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:52.270316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:47865)
0926 03:03:52.270746 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:52.270836 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260926 03:03:52.720091 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:52.719364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:47345)
0926 03:03:52.719846 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:52.719933 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
I20260926 03:03:52.834409 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:52.833695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:58753)
0926 03:03:52.834129 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:52.834237 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":222,"threads_started":1}
I20260926 03:03:53.272464 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:53.271785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:45707)
0926 03:03:53.272230 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:53.272304 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":241,"threads_started":1}
I20260926 03:03:53.721733 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:53.720956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:48225)
0926 03:03:53.721498 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:53.721572 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":359,"threads_started":1}
I20260926 03:03:53.836004 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:53.835261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:46639)
0926 03:03:53.835766 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:53.835840 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":292,"threads_started":1}
I20260926 03:03:54.274008 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:54.273334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:41425)
0926 03:03:54.273738 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:54.273811 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":188,"threads_started":1}
I20260926 03:03:54.723235 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:54.722568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:38867)
0926 03:03:54.722964 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:54.723038 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260926 03:03:54.837742 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:54.836910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:40887)
0926 03:03:54.837413 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:54.837551 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260926 03:03:55.275549 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:55.274877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:46555)
0926 03:03:55.275294 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:55.275375 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260926 03:03:55.724810 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:55.724160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:41559)
0926 03:03:55.724555 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:55.724626 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":200,"threads_started":1}
I20260926 03:03:55.839648 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:55.838757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:36721)
0926 03:03:55.839297 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:55.839451 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
I20260926 03:03:56.277072 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:56.276347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:55111)
0926 03:03:56.276740 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:56.276819 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":212,"threads_started":1}
I20260926 03:03:56.726487 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:56.725657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:37967)
0926 03:03:56.726186 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:56.726269 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":316,"threads_started":1}
I20260926 03:03:56.841468 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:56.840657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:42435)
0926 03:03:56.841167 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:56.841244 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":202,"threads_started":1}
I20260926 03:03:57.278703 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:57.277950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:47841)
0926 03:03:57.278417 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:57.278493 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":198,"threads_started":1}
I20260926 03:03:57.728194 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:57.727442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:42983)
0926 03:03:57.727912 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:57.727989 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":209,"threads_started":1}
I20260926 03:03:57.843110 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:57.842420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:49733)
0926 03:03:57.842861 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:57.842945 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
I20260926 03:03:58.280429 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:58.279598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:34887)
0926 03:03:58.280140 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:58.280264 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":319,"threads_started":1}
I20260926 03:03:58.729748 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:58.729113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:46079)
0926 03:03:58.729512 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:58.729586 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":209,"threads_started":1}
I20260926 03:03:58.844676 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:58.843954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:56713)
0926 03:03:58.844431 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:58.844508 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":230,"threads_started":1}
I20260926 03:03:59.282014 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:59.281345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:55417)
0926 03:03:59.281774 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:59.281848 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":208,"threads_started":1}
I20260926 03:03:59.731210 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:59.730546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:60993)
0926 03:03:59.730971 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:59.731045 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":211,"threads_started":1}
I20260926 03:03:59.846220 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:03:59.845558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:49037)
0926 03:03:59.845978 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 03:03:59.846057 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":242,"threads_started":1}
I20260926 03:04:00.283625 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:00.282975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:48129)
0926 03:04:00.283396 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:00.283473 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":236,"threads_started":1}
I20260926 03:04:00.732776 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:00.732037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:59981)
0926 03:04:00.732541 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:00.732615 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":319,"threads_started":1}
I20260926 03:04:00.847839 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:00.847104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:58695)
0926 03:04:00.847565 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:00.847647 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
I20260926 03:04:01.285336 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:01.284566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:40461)
0926 03:04:01.285059 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:01.285157 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":217,"threads_started":1}
I20260926 03:04:01.734402 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:01.733711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:44415)
0926 03:04:01.734135 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:01.734220 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260926 03:04:01.849555 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:01.848802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:44569)
0926 03:04:01.849316 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:01.849400 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":297,"threads_started":1}
I20260926 03:04:02.286928 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:02.286269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:56009)
0926 03:04:02.286688 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:02.286759 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":229,"threads_started":1}
I20260926 03:04:02.736088 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:02.735414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:42405)
0926 03:04:02.735838 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:02.735938 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":232,"threads_started":1}
I20260926 03:04:02.851466 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:02.850479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:43367)
0926 03:04:02.851068 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:02.851177 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":267,"threads_started":1}
I20260926 03:04:03.288899 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:03.287973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:34541)
0926 03:04:03.288465 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:03.288598 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":216,"threads_started":1}
I20260926 03:04:03.737946 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:03.736996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:38635)
0926 03:04:03.737611 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:03.737696 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":320,"threads_started":1}
I20260926 03:04:03.853214 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:03.852316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:58335)
0926 03:04:03.852817 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:03.852926 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":256,"threads_started":1}
I20260926 03:04:04.290412 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:04.289773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:56039)
0926 03:04:04.290154 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:04.290230 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":188,"threads_started":1}
I20260926 03:04:04.739367 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:04.738698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:33573)
0926 03:04:04.739105 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:04.739180 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":219,"threads_started":1}
I20260926 03:04:04.854935 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:04.854211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.1:38129)
0926 03:04:04.854670 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:04.854752 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":271,"threads_started":1}
I20260926 03:04:05.292249 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:05.291441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.2:34985)
0926 03:04:05.291980 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:05.292071 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
I20260926 03:04:05.722450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 16997
I20260926 03:04:05.741043 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:05.723638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.0.97.1:41445)
0926 03:04:05.724379 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:05.724403 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:05.724455 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:04:05.726243 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:05.726253 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:05.726690 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:05.727070 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:05.728532 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:05.729828 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:05.731076 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:05.731768 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:05.731966 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:05.734555 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:05.734563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:05.734578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:05.734612 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:05.737638 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:05.738359 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:05.738372 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:05.738379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:05.738541 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:05.738950 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:05.738954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:05.738956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:05.739479 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:05.739814 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:05.740290 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 03:04:05.740712 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":328,"threads_started":1}
I20260926 03:04:05.741374 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:05.740504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37913 (local address 127.0.97.3:50079)
0926 03:04:05.741074 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:05.741160 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":280,"threads_started":1}
I20260926 03:04:05.742527 17160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:05.825956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 16997
I20260926 03:04:05.842661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 17168
I20260926 03:04:05.859717 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:05.843626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.0.97.2:43063)
0926 03:04:05.844175 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:05.844188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:05.844225 (+ 37us) server_negotiation.cc:407] Connection header received
0926 03:04:05.845881 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:05.845891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:05.846330 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:05.846618 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:05.847930 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:05.849245 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:05.850421 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:05.851144 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:05.851275 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:05.853886 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:05.853894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:05.853908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:05.853942 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:05.856885 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:05.857530 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:05.857537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:05.857544 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:05.857703 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:05.858021 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:05.858024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:05.858026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:05.858561 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:05.858779 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:05.859100 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:04:05.859436 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":289,"threads_started":1}
I20260926 03:04:05.861718 17291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:05.960407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 17168
I20260926 03:04:05.977828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 17299
I20260926 03:04:05.994979 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:05.978850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.0.97.3:40987)
0926 03:04:05.979532 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:05.979545 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:05.979587 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:04:05.981263 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:05.981273 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:05.981716 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:05.982026 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:05.983450 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:05.984639 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:05.985897 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:05.986538 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:05.986672 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:05.989201 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:05.989208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:05.989223 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:05.989257 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:05.991995 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:05.992657 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:05.992662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:05.992668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:05.992857 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:05.993295 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:05.993298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:05.993300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:05.993792 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:05.994023 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:05.994357 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:04:05.994675 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":357,"threads_started":1}
I20260926 03:04:05.996315 17422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:06.080374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 17299
2026-09-26T03:04:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64275 ms)
[----------] 30 tests from SecurityITest (288151 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:06 dist-test-slave-fccw krb5kdc[17484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:06 dist-test-slave-fccw krb5kdc[17484](info): set up 2 sockets
Sep 26 03:04:06 dist-test-slave-fccw krb5kdc[17484](info): commencing operation
krb5kdc: starting...
W20260926 03:04:07.189073 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:07 dist-test-slave-fccw krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:07.427062 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45063
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:07.755089 17497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:07.755364 17497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:07.755451 17497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:07.762554 17497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:07.762660 17497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:07.762714 17497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:07.762770 17497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:07.762813 17497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:07.772509 17497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:07.773732 17497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:07.774963 17497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:07.786301 17505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:07.786513 17503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:07.786480 17502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:07.787621 17497 server_base.cc:1034] running on GCE node
I20260926 03:04:07.788298 17497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:07.789721 17497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:07.790958 17497 hybrid_clock.cc:648] HybridClock initialized: now 1790391847790919 us; error 48 us; skew 500 ppm
I20260926 03:04:07.791299 17497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:07 dist-test-slave-fccw krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:07.794255 17497 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:04:07.795673 17497 webserver.cc:460] Webserver started at http://127.0.97.62:40269/ using document root <none> and password file <none>
I20260926 03:04:07.796350 17497 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:07.796490 17497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:07.796785 17497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:07.800046 17497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "64d539e0a51d409f83cd36ea69f05ba2"
format_stamp: "Formatted at 2026-09-26 03:04:07 on dist-test-slave-fccw"
server_key: "a59e9e7dc288a6d65ac52cdf3d243ecc"
server_key_iv: "c20efc0cf44772543e74857b80fc241c"
server_key_version: "encryptionkey@0"
I20260926 03:04:07.800765 17497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "64d539e0a51d409f83cd36ea69f05ba2"
format_stamp: "Formatted at 2026-09-26 03:04:07 on dist-test-slave-fccw"
server_key: "a59e9e7dc288a6d65ac52cdf3d243ecc"
server_key_iv: "c20efc0cf44772543e74857b80fc241c"
server_key_version: "encryptionkey@0"
I20260926 03:04:07.805649 17497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:04:07.808753 17512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:07.809991 17497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 03:04:07.810168 17497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "64d539e0a51d409f83cd36ea69f05ba2"
format_stamp: "Formatted at 2026-09-26 03:04:07 on dist-test-slave-fccw"
server_key: "a59e9e7dc288a6d65ac52cdf3d243ecc"
server_key_iv: "c20efc0cf44772543e74857b80fc241c"
server_key_version: "encryptionkey@0"
I20260926 03:04:07.810364 17497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:07.820847 17497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:07.824540 17497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:07.824843 17497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:07.842832 17497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45063
I20260926 03:04:07.842847 17563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45063 every 8 connection(s)
I20260926 03:04:07.844204 17497 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:07.847815 17564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:07.849561 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 17497
I20260926 03:04:07.849772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:07.850449 388 external_mini_cluster.cc:1428] Setting key 8fb4b457e8a28cfc70ef06f5170e14e6
I20260926 03:04:07.858716 17564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2: Bootstrap starting.
I20260926 03:04:07.861761 17564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:07.862728 17564 log.cc:824] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2: Log is configured to *not* fsync() on all Append() calls
Sep 26 03:04:07 dist-test-slave-fccw krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:07.865891 17564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2: No bootstrap required, opened a new log
I20260926 03:04:07.873714 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:07.852399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.0.97.62:45063)
0926 03:04:07.853403 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:07.853415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:07.853454 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:04:07.855079 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:07.855099 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:07.855463 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:07.855857 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:07.857323 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:07.858565 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:07.859968 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:07.860653 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:07.860875 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:07.865535 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:07.865542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:07.865556 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:07.865591 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:07.869432 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:07.870337 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:07.870349 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:07.870357 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:07.870535 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:07.870951 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:07.870956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:07.870959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:07.871485 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:07.871802 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:07.872259 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 03:04:07.872680 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":501,"threads_started":1}
I20260926 03:04:07.875312 17564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d539e0a51d409f83cd36ea69f05ba2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45063 } }
I20260926 03:04:07.875676 17564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:07.875780 17564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d539e0a51d409f83cd36ea69f05ba2, State: Initialized, Role: FOLLOWER
I20260926 03:04:07.876428 17564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d539e0a51d409f83cd36ea69f05ba2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45063 } }
I20260926 03:04:07.876708 17564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:07.876854 17564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:07.877053 17564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:07.879405 17564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d539e0a51d409f83cd36ea69f05ba2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45063 } }
I20260926 03:04:07.879878 17564 leader_election.cc:302] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64d539e0a51d409f83cd36ea69f05ba2; no voters:
I20260926 03:04:07.880445 17564 leader_election.cc:288] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:07.880541 17569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:07.881331 17569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [term 1 LEADER]: Becoming Leader. State: Replica: 64d539e0a51d409f83cd36ea69f05ba2, State: Running, Role: LEADER
I20260926 03:04:07.881829 17569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d539e0a51d409f83cd36ea69f05ba2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45063 } }
I20260926 03:04:07.882500 17564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:07.884563 17571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64d539e0a51d409f83cd36ea69f05ba2. Latest consensus state: current_term: 1 leader_uuid: "64d539e0a51d409f83cd36ea69f05ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d539e0a51d409f83cd36ea69f05ba2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45063 } } }
I20260926 03:04:07.884951 17571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:07.885396 17570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64d539e0a51d409f83cd36ea69f05ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64d539e0a51d409f83cd36ea69f05ba2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45063 } } }
I20260926 03:04:07.885674 17570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:07.887683 17577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:07.891965 17577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:07.902158 17577 catalog_manager.cc:1347] Generated new cluster ID: c0acbdba989b4957af17ed870db3c5b6
I20260926 03:04:07.902266 17577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:07.918741 17577 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:07.920027 17577 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:07.935611 17577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2: Generated new TSK 0
I20260926 03:04:07.936574 17577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:04:07.995980 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45063
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:08.320111 17590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:08.320370 17590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:08.320466 17590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:08.327704 17590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:08.327823 17590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:08.327992 17590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:04:08.337705 17590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:08.338867 17590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:08.340124 17590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:08.353217 17595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:08.353160 17596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:08.353710 17598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:08.354216 17590 server_base.cc:1034] running on GCE node
I20260926 03:04:08.354803 17590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:08.355743 17590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:08.356971 17590 hybrid_clock.cc:648] HybridClock initialized: now 1790391848356939 us; error 52 us; skew 500 ppm
I20260926 03:04:08.357359 17590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:08 dist-test-slave-fccw krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:08.360325 17590 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:04:08.361865 17590 webserver.cc:460] Webserver started at http://127.0.97.1:34229/ using document root <none> and password file <none>
I20260926 03:04:08.362561 17590 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:08.362709 17590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:08.363036 17590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:08.366074 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "b4a509a2be2c4453aad6e9925f40db08"
format_stamp: "Formatted at 2026-09-26 03:04:08 on dist-test-slave-fccw"
server_key: "30e967a86226e91c553ea06bfedea647"
server_key_iv: "21b30ef030b746300b10c790eee60073"
server_key_version: "encryptionkey@0"
I20260926 03:04:08.366974 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b4a509a2be2c4453aad6e9925f40db08"
format_stamp: "Formatted at 2026-09-26 03:04:08 on dist-test-slave-fccw"
server_key: "30e967a86226e91c553ea06bfedea647"
server_key_iv: "21b30ef030b746300b10c790eee60073"
server_key_version: "encryptionkey@0"
I20260926 03:04:08.371991 17590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:04:08.375564 17605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:08.376904 17590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:04:08.377102 17590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "b4a509a2be2c4453aad6e9925f40db08"
format_stamp: "Formatted at 2026-09-26 03:04:08 on dist-test-slave-fccw"
server_key: "30e967a86226e91c553ea06bfedea647"
server_key_iv: "21b30ef030b746300b10c790eee60073"
server_key_version: "encryptionkey@0"
I20260926 03:04:08.377312 17590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:08.390954 17590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:08.394665 17590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:08.394939 17590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:08.395818 17590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:08.397111 17590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:08.397207 17590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:08.397357 17590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:08.397432 17590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:08.429896 17590 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40949
I20260926 03:04:08.429939 17717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40949 every 8 connection(s)
I20260926 03:04:08.431140 17590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:04:08.438194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 17590
I20260926 03:04:08.438426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:04:08.439172 388 external_mini_cluster.cc:1428] Setting key 1ac34d82480cc3367f148a41d4f48c6d
Sep 26 03:04:08 dist-test-slave-fccw krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:08.452764 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:08.433651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41019 (local address 127.0.97.62:45063)
0926 03:04:08.434129 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:08.434134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:08.434972 (+ 838us) server_negotiation.cc:407] Connection header received
0926 03:04:08.436846 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:08.436851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:08.436945 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:08.437108 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:08.438937 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.439728 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.441744 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.442458 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.442601 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:08.446113 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:08.446123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:08.446131 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:08.446176 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:08.449599 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:08.450545 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:08.450549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:08.450551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:08.450665 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:08.451127 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:08.451130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:08.451132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:08.451373 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:08.451492 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:08.452336 (+ 844us) server_negotiation.cc:299] Negotiation successful
0926 03:04:08.452500 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":266,"threads_started":1}
I20260926 03:04:08.454306 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:08.434128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45063 (local address 127.0.97.1:41019)
0926 03:04:08.434793 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:08.434835 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:08.436586 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:08.437326 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:08.437335 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:08.437945 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:08.438757 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:08.438783 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.439891 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.439899 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:08.441598 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:08.441609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.442667 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.442671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:08.442858 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:08.443510 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:08.443531 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:08.445873 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:08.449737 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:08.449748 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:08.449766 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:08.450420 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:08.450769 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:08.450777 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:08.450783 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:08.451021 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:08.451494 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:08.451504 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:08.451945 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 03:04:08.452246 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 03:04:08.452810 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":284,"threads_started":1}
I20260926 03:04:08.455933 17718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45063
I20260926 03:04:08.456256 17718 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:08.456981 17718 heartbeater.cc:511] Master 127.0.97.62:45063 requested a full tablet report, sending...
I20260926 03:04:08.459293 17529 ts_manager.cc:194] Registered new tserver with Master: b4a509a2be2c4453aad6e9925f40db08 (127.0.97.1:40949)
I20260926 03:04:08.461534 17529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41019
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:04:08.495571 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45063
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:08.814880 17724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:08.815245 17724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:08.815407 17724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:08.822583 17724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:08.822738 17724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:08.822932 17724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:04:08.832623 17724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:08.833961 17724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:08.835240 17724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:08.848004 17730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:08.848165 17729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:08.848281 17732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:08.849584 17724 server_base.cc:1034] running on GCE node
I20260926 03:04:08.850200 17724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:08.851295 17724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:08.852525 17724 hybrid_clock.cc:648] HybridClock initialized: now 1790391848852464 us; error 53 us; skew 500 ppm
I20260926 03:04:08.852869 17724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:08 dist-test-slave-fccw krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:08.856489 17724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:04:08.858008 17724 webserver.cc:460] Webserver started at http://127.0.97.2:33269/ using document root <none> and password file <none>
I20260926 03:04:08.858688 17724 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:08.858829 17724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:08.859133 17724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:08.862106 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "62b86eb7e6fd4a9da28ff469e67ecd45"
format_stamp: "Formatted at 2026-09-26 03:04:08 on dist-test-slave-fccw"
server_key: "d855fc623e7f16cf1b2a8399c88369e1"
server_key_iv: "b7020022cf657fdfa9ccf48fa55c3af3"
server_key_version: "encryptionkey@0"
I20260926 03:04:08.862867 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "62b86eb7e6fd4a9da28ff469e67ecd45"
format_stamp: "Formatted at 2026-09-26 03:04:08 on dist-test-slave-fccw"
server_key: "d855fc623e7f16cf1b2a8399c88369e1"
server_key_iv: "b7020022cf657fdfa9ccf48fa55c3af3"
server_key_version: "encryptionkey@0"
I20260926 03:04:08.867962 17724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:04:08.871762 17739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:08.873485 17724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 03:04:08.873658 17724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "62b86eb7e6fd4a9da28ff469e67ecd45"
format_stamp: "Formatted at 2026-09-26 03:04:08 on dist-test-slave-fccw"
server_key: "d855fc623e7f16cf1b2a8399c88369e1"
server_key_iv: "b7020022cf657fdfa9ccf48fa55c3af3"
server_key_version: "encryptionkey@0"
I20260926 03:04:08.873906 17724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:08.889436 17724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:08.893352 17724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:08.893627 17724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:08.894572 17724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:08.895849 17724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:08.895918 17724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:08.896080 17724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:08.896134 17724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:08.928261 17724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40039
I20260926 03:04:08.928306 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40039 every 8 connection(s)
I20260926 03:04:08.929587 17724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:04:08.937987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 17724
I20260926 03:04:08.938184 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:04:08.938776 388 external_mini_cluster.cc:1428] Setting key f27fd64814553ce53100a9b3e2a943cb
Sep 26 03:04:08 dist-test-slave-fccw krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:08.951176 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:08.932129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60933 (local address 127.0.97.62:45063)
0926 03:04:08.932312 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:08.932316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:08.933550 (+ 1234us) server_negotiation.cc:407] Connection header received
0926 03:04:08.935393 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:08.935398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:08.935493 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:08.935606 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:08.937156 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.938003 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.940169 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.940699 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.940791 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:08.944250 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:08.944259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:08.944266 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:08.944309 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:08.947913 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:08.948787 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:08.948795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:08.948802 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:08.948979 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:08.949520 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:08.949523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:08.949525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:08.949777 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:08.949904 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:08.950749 (+ 845us) server_negotiation.cc:299] Negotiation successful
0926 03:04:08.950930 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:04:08.952724 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:08.932633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45063 (local address 127.0.97.2:60933)
0926 03:04:08.933387 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:08.933424 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:08.935118 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:08.935856 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:08.935864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:08.936359 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:08.936929 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:08.936946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.938200 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.938207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:08.940037 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:08.940051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:08.940916 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:08.940922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:08.941182 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:08.941856 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:08.941878 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:08.944012 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:08.948076 (+ 4064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:08.948089 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:08.948107 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:08.948649 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:08.949127 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:08.949135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:08.949141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:08.949403 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:08.949903 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:08.949914 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:08.950366 (+ 452us) client_negotiation.cc:769] Sending connection context
0926 03:04:08.950673 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 03:04:08.951247 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":326,"threads_started":1}
I20260926 03:04:08.954205 17852 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45063
I20260926 03:04:08.954524 17852 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:08.955348 17852 heartbeater.cc:511] Master 127.0.97.62:45063 requested a full tablet report, sending...
I20260926 03:04:08.956761 17529 ts_manager.cc:194] Registered new tserver with Master: 62b86eb7e6fd4a9da28ff469e67ecd45 (127.0.97.2:40039)
I20260926 03:04:08.957618 17529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60933
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:04:08.996110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45063
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:09.323202 17857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:09.323463 17857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:09.323611 17857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:09.331115 17857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:09.331233 17857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:09.331413 17857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:04:09.340675 17857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:09.341910 17857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:09.343158 17857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:09.355872 17865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:09.355942 17862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:09.355940 17863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:09.356696 17857 server_base.cc:1034] running on GCE node
I20260926 03:04:09.357309 17857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:09.358232 17857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:09.359423 17857 hybrid_clock.cc:648] HybridClock initialized: now 1790391849359376 us; error 37 us; skew 500 ppm
I20260926 03:04:09.359723 17857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:09 dist-test-slave-fccw krb5kdc[17484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:09.362866 17857 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:04:09.364360 17857 webserver.cc:460] Webserver started at http://127.0.97.3:34301/ using document root <none> and password file <none>
I20260926 03:04:09.365116 17857 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:09.365257 17857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:09.365566 17857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:09.368577 17857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "4ad501b337fc46848434952c3763a6c2"
format_stamp: "Formatted at 2026-09-26 03:04:09 on dist-test-slave-fccw"
server_key: "2f34a59e63f6da1dd8c395ff6ef2e566"
server_key_iv: "23d17f158de88098f15875c59d488f2d"
server_key_version: "encryptionkey@0"
I20260926 03:04:09.369396 17857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4ad501b337fc46848434952c3763a6c2"
format_stamp: "Formatted at 2026-09-26 03:04:09 on dist-test-slave-fccw"
server_key: "2f34a59e63f6da1dd8c395ff6ef2e566"
server_key_iv: "23d17f158de88098f15875c59d488f2d"
server_key_version: "encryptionkey@0"
I20260926 03:04:09.374239 17857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:04:09.377485 17872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:09.378881 17857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:04:09.379051 17857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "4ad501b337fc46848434952c3763a6c2"
format_stamp: "Formatted at 2026-09-26 03:04:09 on dist-test-slave-fccw"
server_key: "2f34a59e63f6da1dd8c395ff6ef2e566"
server_key_iv: "23d17f158de88098f15875c59d488f2d"
server_key_version: "encryptionkey@0"
I20260926 03:04:09.379251 17857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:09.401403 17857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:09.405462 17857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:09.405738 17857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:09.406682 17857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:09.407948 17857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:09.408052 17857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:09.408205 17857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:09.408275 17857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:09.440852 17857 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38845
I20260926 03:04:09.440941 17984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38845 every 8 connection(s)
I20260926 03:04:09.442119 17857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:04:09.449158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 17857
I20260926 03:04:09.449363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:04:09.449954 388 external_mini_cluster.cc:1428] Setting key 051e8fb449dcf037f2e9bfd544d8cf4c
Sep 26 03:04:09 dist-test-slave-fccw krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:09.461405 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.444525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50623 (local address 127.0.97.62:45063)
0926 03:04:09.444701 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.444705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.445875 (+ 1170us) server_negotiation.cc:407] Connection header received
0926 03:04:09.447704 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.447710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.447798 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.447912 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:09.449442 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.450535 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.452273 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.452731 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.452862 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.456181 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:09.456185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:09.456189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:09.456219 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:09.458893 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:09.459508 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:09.459512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:09.459514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:09.459635 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:09.459971 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:09.459973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:09.459975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:09.460216 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:09.460334 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.460907 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.461137 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:04:09.462306 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.444972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45063 (local address 127.0.97.3:50623)
0926 03:04:09.445710 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:09.445749 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:09.447439 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:09.448131 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:09.448139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:09.448618 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:09.449230 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.449249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.450689 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.450696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.452099 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.452111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.452978 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.452987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.453239 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.453864 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:09.453892 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:09.455974 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:09.459015 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:09.459021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:09.459030 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:09.459398 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:09.459720 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:09.459723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:09.459725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:09.459879 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:09.460316 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:09.460321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:09.460600 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 03:04:09.460803 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 03:04:09.461236 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":267,"threads_started":1}
I20260926 03:04:09.463814 17985 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45063
I20260926 03:04:09.464145 17985 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:09.464555 17718 heartbeater.cc:503] Master 127.0.97.62:45063 was elected leader, sending a full tablet report...
I20260926 03:04:09.465013 17985 heartbeater.cc:511] Master 127.0.97.62:45063 requested a full tablet report, sending...
I20260926 03:04:09.466356 17529 ts_manager.cc:194] Registered new tserver with Master: 4ad501b337fc46848434952c3763a6c2 (127.0.97.3:38845)
I20260926 03:04:09.467056 17529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50623
I20260926 03:04:09.477380 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:04:09.494174 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.481970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.0.97.62:45063)
0926 03:04:09.482167 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.482173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.482635 (+ 462us) server_negotiation.cc:407] Connection header received
0926 03:04:09.482835 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.482839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.482907 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.482989 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:09.484542 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.485298 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.486522 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.486898 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.486986 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.489297 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:09.489301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:09.489305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:09.489333 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:09.491968 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:09.492548 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:09.492552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:09.492554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:09.492666 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:09.492987 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:09.492990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:09.493012 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:09.493299 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:09.493473 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.493764 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.493920 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:04:09.497978 17529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:04:09.504802 17529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:04:09.527899 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.517421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53770 (local address 127.0.97.2:40039)
0926 03:04:09.518427 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.518438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.519586 (+ 1148us) server_negotiation.cc:407] Connection header received
0926 03:04:09.519965 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.519977 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.520261 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.520436 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:09.521121 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.522637 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.525185 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.526488 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.526612 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.526806 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.527392 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.527547 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":1348,"threads_started":1}
I20260926 03:04:09.527908 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.516883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40039 (local address 127.0.0.1:53770)
0926 03:04:09.519452 (+ 2569us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:09.519475 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:09.519711 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:09.520640 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:09.520647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:09.520692 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:09.520928 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.520936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.522822 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.522827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.524985 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.525050 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.526938 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.526944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.527141 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.527178 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:04:09.527325 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 03:04:09.527512 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2417,"thread_start_us":713,"threads_started":1}
I20260926 03:04:09.528565 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.516627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:33952)
0926 03:04:09.517651 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:09.517697 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:09.517882 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:09.521526 (+ 3644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:09.521532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:09.521558 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:09.521961 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.521971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.523850 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.523855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.525854 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.525868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.527723 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.527730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.527926 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.527956 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:04:09.528020 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 03:04:09.528140 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":21,"spinlock_wait_cycles":8448,"thread_start_us":2514,"threads_started":1}
I20260926 03:04:09.529635 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.516925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.97.1:40949)
0926 03:04:09.521085 (+ 4160us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.521091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.521111 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:04:09.521179 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.521184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.521349 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.521497 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:09.522147 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.523718 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.526031 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.527498 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.528809 (+ 1311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.529071 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.529138 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.529306 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3978,"thread_start_us":3795,"threads_started":1}
I20260926 03:04:09.529723 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.518526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38845 (local address 127.0.0.1:33584)
0926 03:04:09.520408 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:09.520430 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:09.521715 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:09.522274 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:09.522278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:09.522309 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:09.522634 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.522645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.524580 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.524586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.526742 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.526756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.528911 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.528916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.529110 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.529139 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 03:04:09.529224 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:04:09.529347 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1701,"mutex_wait_us":22,"spinlock_wait_cycles":768,"thread_start_us":1553,"threads_started":1}
I20260926 03:04:09.530422 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.518886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.0.97.3:38845)
0926 03:04:09.519891 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.519904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.520538 (+ 634us) server_negotiation.cc:407] Connection header received
0926 03:04:09.521905 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.521910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.522085 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.522274 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:09.522825 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.524428 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.526928 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.528311 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.528439 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.528793 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.529968 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.530118 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":670,"threads_started":1}
I20260926 03:04:09.531898 17787 tablet_service.cc:1467] Processing CreateTablet for tablet 69b3ac3f62fc4f81a6d45f5090660fe0 (DEFAULT_TABLE table=test-table [id=249f7ebb45324a2c8b780ee2e8d03a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:09.532866 17653 tablet_service.cc:1467] Processing CreateTablet for tablet 69b3ac3f62fc4f81a6d45f5090660fe0 (DEFAULT_TABLE table=test-table [id=249f7ebb45324a2c8b780ee2e8d03a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:09.533450 17787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b3ac3f62fc4f81a6d45f5090660fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:09.534049 17653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b3ac3f62fc4f81a6d45f5090660fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:09.534054 17920 tablet_service.cc:1467] Processing CreateTablet for tablet 69b3ac3f62fc4f81a6d45f5090660fe0 (DEFAULT_TABLE table=test-table [id=249f7ebb45324a2c8b780ee2e8d03a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:09.535622 17920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b3ac3f62fc4f81a6d45f5090660fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:09.544984 18003 tablet_bootstrap.cc:492] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: Bootstrap starting.
I20260926 03:04:09.548077 18004 tablet_bootstrap.cc:492] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Bootstrap starting.
I20260926 03:04:09.548653 18003 tablet_bootstrap.cc:654] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:09.549582 18005 tablet_bootstrap.cc:492] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Bootstrap starting.
I20260926 03:04:09.551090 18003 log.cc:824] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:09.551965 18004 tablet_bootstrap.cc:654] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:09.552908 18005 tablet_bootstrap.cc:654] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:09.553076 18004 log.cc:824] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:09.554087 18005 log.cc:824] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:09.554342 18003 tablet_bootstrap.cc:492] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: No bootstrap required, opened a new log
I20260926 03:04:09.554611 18003 ts_tablet_manager.cc:1397] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 03:04:09.555959 18004 tablet_bootstrap.cc:492] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: No bootstrap required, opened a new log
I20260926 03:04:09.556200 18004 ts_tablet_manager.cc:1397] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 03:04:09.556550 18005 tablet_bootstrap.cc:492] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: No bootstrap required, opened a new log
I20260926 03:04:09.556793 18005 ts_tablet_manager.cc:1397] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 03:04:09.564067 18005 raft_consensus.cc:348] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.564445 18005 raft_consensus.cc:374] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:09.564601 18005 raft_consensus.cc:729] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad501b337fc46848434952c3763a6c2, State: Initialized, Role: FOLLOWER
I20260926 03:04:09.565245 18005 consensus_queue.cc:260] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.565274 18003 raft_consensus.cc:348] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.565696 18003 raft_consensus.cc:374] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:09.565848 18003 raft_consensus.cc:729] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4a509a2be2c4453aad6e9925f40db08, State: Initialized, Role: FOLLOWER
I20260926 03:04:09.566380 17985 heartbeater.cc:503] Master 127.0.97.62:45063 was elected leader, sending a full tablet report...
I20260926 03:04:09.566560 18003 consensus_queue.cc:260] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.566612 18004 raft_consensus.cc:348] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.566977 18004 raft_consensus.cc:374] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:09.567041 18005 ts_tablet_manager.cc:1428] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
I20260926 03:04:09.567092 18004 raft_consensus.cc:729] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b86eb7e6fd4a9da28ff469e67ecd45, State: Initialized, Role: FOLLOWER
I20260926 03:04:09.567811 18004 consensus_queue.cc:260] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.568648 18003 ts_tablet_manager.cc:1428] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260926 03:04:09.569548 17852 heartbeater.cc:503] Master 127.0.97.62:45063 was elected leader, sending a full tablet report...
I20260926 03:04:09.571085 18004 ts_tablet_manager.cc:1428] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260926 03:04:09.681546 17853 tablet.cc:2369] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:04:09.683336 17719 tablet.cc:2369] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:04:09.693766 17986 tablet.cc:2369] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:04:09.914203 18011 raft_consensus.cc:482] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:04:09.914587 18011 raft_consensus.cc:504] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.916432 18011 leader_election.cc:288] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4a509a2be2c4453aad6e9925f40db08 (127.0.97.1:40949), 4ad501b337fc46848434952c3763a6c2 (127.0.97.3:38845)
I20260926 03:04:09.923261 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.917141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48725 (local address 127.0.97.3:38845)
0926 03:04:09.917316 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.917321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.917956 (+ 635us) server_negotiation.cc:407] Connection header received
0926 03:04:09.918114 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.918118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.918202 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.918289 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:09.918972 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.919655 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.921491 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.922287 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.922388 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.922512 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.922856 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.922986 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:04:09.923275 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.917036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35163 (local address 127.0.97.1:40949)
0926 03:04:09.917239 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.917245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.917815 (+ 570us) server_negotiation.cc:407] Connection header received
0926 03:04:09.918038 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.918045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.918127 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.918192 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:09.918855 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.919657 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.921568 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.922331 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.922501 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.922595 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.922906 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.923027 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:04:09.923440 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.916962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38845 (local address 127.0.97.2:48725)
0926 03:04:09.917812 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:09.917839 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:09.917984 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:09.918322 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:09.918327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:09.918358 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:09.918811 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.918829 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.919791 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.919798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.921336 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.921351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.922520 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.922526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.922697 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.922724 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 03:04:09.922797 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 03:04:09.922942 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":10,"thread_start_us":499,"threads_started":1}
I20260926 03:04:09.923485 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.916754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.97.2:35163)
0926 03:04:09.917680 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:09.917706 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:09.917860 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:09.918244 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:09.918252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:09.918280 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:09.918706 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.918716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.919793 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.919801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.921415 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:09.921433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.922581 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.922586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:09.922716 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.922738 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:04:09.922863 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 03:04:09.922984 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":68,"spinlock_wait_cycles":23936,"thread_start_us":492,"threads_started":1}
I20260926 03:04:09.924757 17940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b3ac3f62fc4f81a6d45f5090660fe0" candidate_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad501b337fc46848434952c3763a6c2" is_pre_election: true
I20260926 03:04:09.924757 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b3ac3f62fc4f81a6d45f5090660fe0" candidate_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a509a2be2c4453aad6e9925f40db08" is_pre_election: true
I20260926 03:04:09.925336 17940 raft_consensus.cc:2454] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62b86eb7e6fd4a9da28ff469e67ecd45 in term 0.
I20260926 03:04:09.925340 17673 raft_consensus.cc:2454] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62b86eb7e6fd4a9da28ff469e67ecd45 in term 0.
I20260926 03:04:09.926043 17740 leader_election.cc:302] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ad501b337fc46848434952c3763a6c2, 62b86eb7e6fd4a9da28ff469e67ecd45, b4a509a2be2c4453aad6e9925f40db08; no voters:
I20260926 03:04:09.926648 18011 raft_consensus.cc:2781] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:04:09.926817 18011 raft_consensus.cc:482] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:04:09.926935 18011 raft_consensus.cc:3037] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:09.929390 18011 raft_consensus.cc:504] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.930012 18011 leader_election.cc:288] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [CANDIDATE]: Term 1 election: Requested vote from peers b4a509a2be2c4453aad6e9925f40db08 (127.0.97.1:40949), 4ad501b337fc46848434952c3763a6c2 (127.0.97.3:38845)
I20260926 03:04:09.930485 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b3ac3f62fc4f81a6d45f5090660fe0" candidate_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4a509a2be2c4453aad6e9925f40db08"
I20260926 03:04:09.930572 17940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b3ac3f62fc4f81a6d45f5090660fe0" candidate_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ad501b337fc46848434952c3763a6c2"
I20260926 03:04:09.930722 17673 raft_consensus.cc:3037] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:09.930784 17940 raft_consensus.cc:3037] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:09.933145 17673 raft_consensus.cc:2454] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62b86eb7e6fd4a9da28ff469e67ecd45 in term 1.
I20260926 03:04:09.933568 17743 leader_election.cc:302] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62b86eb7e6fd4a9da28ff469e67ecd45, b4a509a2be2c4453aad6e9925f40db08; no voters:
I20260926 03:04:09.933853 18011 raft_consensus.cc:2781] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:09.933960 17940 raft_consensus.cc:2454] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62b86eb7e6fd4a9da28ff469e67ecd45 in term 1.
I20260926 03:04:09.937279 18011 raft_consensus.cc:686] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 1 LEADER]: Becoming Leader. State: Replica: 62b86eb7e6fd4a9da28ff469e67ecd45, State: Running, Role: LEADER
I20260926 03:04:09.937983 18011 consensus_queue.cc:237] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } }
I20260926 03:04:09.942565 17527 catalog_manager.cc:5495] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62b86eb7e6fd4a9da28ff469e67ecd45 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62b86eb7e6fd4a9da28ff469e67ecd45" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 } health_report { overall_health: UNKNOWN } } }
I20260926 03:04:09.984954 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:09.979307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.0.97.2:40039)
0926 03:04:09.979515 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:09.979519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:09.979537 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:04:09.979727 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:09.979731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:09.979813 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:09.979936 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:04:09.980261 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.981030 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.982633 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:09.983026 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:09.983167 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:09.983773 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:04:09.984372 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:04:09.984488 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:09.984547 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 03:04:09.984611 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:04:09.993203 17940 raft_consensus.cc:1261] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 1 FOLLOWER]: Refusing update from remote peer 62b86eb7e6fd4a9da28ff469e67ecd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:09.993203 17673 raft_consensus.cc:1261] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 1 FOLLOWER]: Refusing update from remote peer 62b86eb7e6fd4a9da28ff469e67ecd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:09.994135 18014 consensus_queue.cc:1035] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4a509a2be2c4453aad6e9925f40db08" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:09.994490 18011 consensus_queue.cc:1035] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ad501b337fc46848434952c3763a6c2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:10.008273 18017 mvcc.cc:205] Tried to move back new op lower bound from 7333445017562497024 to 7333445017349873664. Current Snapshot: MvccSnapshot[applied={T|T < 7333445017562497024}]
I20260926 03:04:10.020314 17527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:10.020620 17527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:10.023882 17527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64d539e0a51d409f83cd36ea69f05ba2: Sending DeleteTablet for 3 replicas of tablet 69b3ac3f62fc4f81a6d45f5090660fe0
I20260926 03:04:10.025058 17653 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b3ac3f62fc4f81a6d45f5090660fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33952
I20260926 03:04:10.025290 17787 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b3ac3f62fc4f81a6d45f5090660fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53770
I20260926 03:04:10.025332 17920 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b3ac3f62fc4f81a6d45f5090660fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33584
I20260926 03:04:10.026134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 17590
I20260926 03:04:10.036449 18027 tablet_replica.cc:330] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: stopping tablet replica
I20260926 03:04:10.037020 18027 raft_consensus.cc:2229] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:10.037667 18027 raft_consensus.cc:2258] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:10.038496 18029 tablet_replica.cc:330] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: stopping tablet replica
I20260926 03:04:10.038910 18029 raft_consensus.cc:2229] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:10.039379 18029 raft_consensus.cc:2258] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:10.041682 18029 ts_tablet_manager.cc:1905] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:10.046562 18027 ts_tablet_manager.cc:1905] T 69b3ac3f62fc4f81a6d45f5090660fe0 P b4a509a2be2c4453aad6e9925f40db08: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:04:10 dist-test-slave-fccw krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:04:10.053524 18028 tablet_replica.cc:330] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: stopping tablet replica
I20260926 03:04:10.053977 18028 raft_consensus.cc:2229] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:04:10.054761 18028 raft_consensus.cc:2258] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:10.057668 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.027809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.97.1:40949)
0926 03:04:10.028035 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:10.028045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:10.038288 (+ 10243us) server_negotiation.cc:407] Connection header received
0926 03:04:10.038525 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:10.038533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:10.038658 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:10.038799 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:10.040602 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:10.041735 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:10.043670 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:10.044329 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:10.044422 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:10.050319 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:10.050338 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:10.050358 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:10.050400 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:10.054445 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:10.055346 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:10.055356 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:10.055361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:10.055533 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:10.055954 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:10.055959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:10.055962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:10.056404 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:10.056698 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:10.057073 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 03:04:10.057300 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:04:10.058424 17713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:10.059841 18029 ts_tablet_manager.cc:1918] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:10.060038 18029 log.cc:1196] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/69b3ac3f62fc4f81a6d45f5090660fe0
I20260926 03:04:10.060634 18029 ts_tablet_manager.cc:1939] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 4ad501b337fc46848434952c3763a6c2: Deleting consensus metadata
I20260926 03:04:10.062394 18028 ts_tablet_manager.cc:1905] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:10.066691 17513 catalog_manager.cc:4841] TS 4ad501b337fc46848434952c3763a6c2 (127.0.97.3:38845): tablet 69b3ac3f62fc4f81a6d45f5090660fe0 (table test-table [id=249f7ebb45324a2c8b780ee2e8d03a59]) successfully deleted
I20260926 03:04:10.068930 18028 ts_tablet_manager.cc:1918] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:10.069135 18028 log.cc:1196] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/69b3ac3f62fc4f81a6d45f5090660fe0
I20260926 03:04:10.069695 18028 ts_tablet_manager.cc:1939] T 69b3ac3f62fc4f81a6d45f5090660fe0 P 62b86eb7e6fd4a9da28ff469e67ecd45: Deleting consensus metadata
I20260926 03:04:10.071666 17513 catalog_manager.cc:4841] TS 62b86eb7e6fd4a9da28ff469e67ecd45 (127.0.97.2:40039): tablet 69b3ac3f62fc4f81a6d45f5090660fe0 (table test-table [id=249f7ebb45324a2c8b780ee2e8d03a59]) successfully deleted
I20260926 03:04:10.160282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 17590
I20260926 03:04:10.178861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 17724
W20260926 03:04:10.178902 17516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40949 (error 108)
I20260926 03:04:10.182642 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.180961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:33974)
0926 03:04:10.182291 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.182413 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1199,"thread_start_us":1074,"threads_started":1}
W20260926 03:04:10.182999 17516 catalog_manager.cc:4584] TS b4a509a2be2c4453aad6e9925f40db08 (127.0.97.1:40949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69b3ac3f62fc4f81a6d45f5090660fe0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Sep 26 03:04:10 dist-test-slave-fccw krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:04:10.198347 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.179890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.0.97.2:40039)
0926 03:04:10.180070 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:10.180076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:10.180097 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:04:10.180302 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:10.180307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:10.180411 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:10.180507 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:10.182612 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:10.183683 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:10.185703 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:10.186101 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:10.186196 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:10.191391 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:10.191409 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:10.191429 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:10.191516 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:10.195489 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:10.196182 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:10.196191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:10.196195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:10.196380 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:10.196719 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:10.196724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:10.196727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:10.197179 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:10.197542 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:10.197705 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:04:10.197926 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:04:10.199083 17847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:10.236774 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.236193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:33978)
0926 03:04:10.236457 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.236574 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 03:04:10.237872 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.237469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:33988)
0926 03:04:10.237632 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.237711 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 03:04:10.298447 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.297856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:33990)
0926 03:04:10.298121 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.298246 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 03:04:10.299508 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.299122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:34004)
0926 03:04:10.299270 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.299358 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 03:04:10.314937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 17724
I20260926 03:04:10.333518 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 17857
Sep 26 03:04:10 dist-test-slave-fccw krb5kdc[17484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:04:10.348101 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.334227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.0.97.3:38845)
0926 03:04:10.334418 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:10.334424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:10.334495 (+ 71us) server_negotiation.cc:407] Connection header received
0926 03:04:10.334700 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:10.334709 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:10.334772 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:10.334893 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:10.336145 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:10.336843 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:10.338181 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:10.338577 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:10.338704 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:10.343113 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:10.343125 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:10.343141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:10.343173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:10.345785 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:10.346446 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:10.346453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:10.346455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:10.346579 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:10.346881 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:10.346884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:10.346886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:10.347187 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:10.347415 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:10.347682 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:04:10.347838 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:04:10.348779 17980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:10.380339 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.379849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:34008)
0926 03:04:10.380101 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.380184 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:04:10.381497 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.381085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:34014)
0926 03:04:10.381284 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.381363 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:04:10.437366 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 17857
I20260926 03:04:10.453959 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 17497
I20260926 03:04:10.455242 17559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:10.565183 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:10.564530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40949 (local address 127.0.0.1:34030)
0926 03:04:10.564862 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:10.565023 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260926 03:04:10.565163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 17497
2026-09-26T03:04:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4484 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T03:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:10Z Disabled control of system clock
I20260926 03:04:10.636363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39547
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:04:10.959141 18040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:10.959443 18040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:10.959525 18040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:10.966711 18040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:10.966817 18040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:10.966856 18040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:10.966914 18040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:10.966959 18040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:10.976255 18040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:10.977463 18040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:10.979184 18040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:10.990972 18045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:10.991122 18048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:10.991129 18046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:10.992195 18040 server_base.cc:1034] running on GCE node
I20260926 03:04:10.992933 18040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:10.994303 18040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:10.995543 18040 hybrid_clock.cc:648] HybridClock initialized: now 1790391850995492 us; error 56 us; skew 500 ppm
I20260926 03:04:10.995913 18040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:10.998224 18040 webserver.cc:460] Webserver started at http://127.0.97.62:36813/ using document root <none> and password file <none>
I20260926 03:04:10.998906 18040 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:10.999040 18040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:10.999362 18040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:11.002281 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "1b572997e7e84c6fbe5360fe810475cc"
format_stamp: "Formatted at 2026-09-26 03:04:10 on dist-test-slave-fccw"
server_key: "126ff344afdb8fa5c42e9bae9e7f5818"
server_key_iv: "45a7c875952708ad420dc469190f232c"
server_key_version: "encryptionkey@0"
I20260926 03:04:11.003053 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b572997e7e84c6fbe5360fe810475cc"
format_stamp: "Formatted at 2026-09-26 03:04:10 on dist-test-slave-fccw"
server_key: "126ff344afdb8fa5c42e9bae9e7f5818"
server_key_iv: "45a7c875952708ad420dc469190f232c"
server_key_version: "encryptionkey@0"
I20260926 03:04:11.007946 18040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 03:04:11.011236 18054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:11.012473 18040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:04:11.012648 18040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "1b572997e7e84c6fbe5360fe810475cc"
format_stamp: "Formatted at 2026-09-26 03:04:10 on dist-test-slave-fccw"
server_key: "126ff344afdb8fa5c42e9bae9e7f5818"
server_key_iv: "45a7c875952708ad420dc469190f232c"
server_key_version: "encryptionkey@0"
I20260926 03:04:11.012852 18040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:11.035804 18040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:11.036621 18040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:11.036911 18040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:11.055346 18040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39547
I20260926 03:04:11.055369 18105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39547 every 8 connection(s)
I20260926 03:04:11.056670 18040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:11.060745 18106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:11.066339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18040
I20260926 03:04:11.066519 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:11.067035 388 external_mini_cluster.cc:1428] Setting key 3845d96e85f1a58fee04b184b4557232
I20260926 03:04:11.072505 18106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc: Bootstrap starting.
I20260926 03:04:11.076393 18106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:11.077436 18106 log.cc:824] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:11.080086 18106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc: No bootstrap required, opened a new log
I20260926 03:04:11.082175 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:11.069046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.0.97.62:39547)
0926 03:04:11.070193 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:11.070207 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:11.070256 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:04:11.072614 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:11.072640 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:11.073280 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:11.073690 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:11.074276 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:11.076178 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:11.077630 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:11.078500 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:11.078698 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:11.079870 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:11.079876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:11.079890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:11.080485 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:11.080588 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:11.080611 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:11.080732 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:11.080953 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:04:11.081252 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":365,"threads_started":1}
I20260926 03:04:11.088300 18106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b572997e7e84c6fbe5360fe810475cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39547 } }
I20260926 03:04:11.088727 18106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:11.088853 18106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b572997e7e84c6fbe5360fe810475cc, State: Initialized, Role: FOLLOWER
I20260926 03:04:11.089550 18106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b572997e7e84c6fbe5360fe810475cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39547 } }
I20260926 03:04:11.089815 18106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:11.089972 18106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:11.090150 18106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:11.092734 18106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b572997e7e84c6fbe5360fe810475cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39547 } }
I20260926 03:04:11.093397 18106 leader_election.cc:302] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b572997e7e84c6fbe5360fe810475cc; no voters:
I20260926 03:04:11.094142 18106 leader_election.cc:288] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:11.094295 18111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:11.095167 18111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [term 1 LEADER]: Becoming Leader. State: Replica: 1b572997e7e84c6fbe5360fe810475cc, State: Running, Role: LEADER
I20260926 03:04:11.095801 18111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b572997e7e84c6fbe5360fe810475cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39547 } }
I20260926 03:04:11.098960 18106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:11.100153 18113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b572997e7e84c6fbe5360fe810475cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b572997e7e84c6fbe5360fe810475cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39547 } } }
I20260926 03:04:11.100267 18112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b572997e7e84c6fbe5360fe810475cc. Latest consensus state: current_term: 1 leader_uuid: "1b572997e7e84c6fbe5360fe810475cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b572997e7e84c6fbe5360fe810475cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39547 } } }
I20260926 03:04:11.100630 18113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:11.100636 18112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:11.102869 18115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:11.108429 18115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:11.117203 18115 catalog_manager.cc:1347] Generated new cluster ID: d33fbb6ebfab4f1aa49d046aa5be25ad
I20260926 03:04:11.117321 18115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:11.131927 18115 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:11.133296 18115 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:11.139215 18115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc: Generated new TSK 0
I20260926 03:04:11.140007 18115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:11.148092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39547
--builtin_ntp_servers=127.0.97.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:04:11.491339 18130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:11.491603 18130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:11.491685 18130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:11.498839 18130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:11.498932 18130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:11.499035 18130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:04:11.508071 18130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:11.509335 18130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:11.511034 18130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:11.524085 18135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:11.523994 18136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:11.524046 18138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:11.525029 18130 server_base.cc:1034] running on GCE node
I20260926 03:04:11.525655 18130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:11.526518 18130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:11.527781 18130 hybrid_clock.cc:648] HybridClock initialized: now 1790391851527707 us; error 63 us; skew 500 ppm
I20260926 03:04:11.528090 18130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:11.536039 18130 webserver.cc:460] Webserver started at http://127.0.97.1:45837/ using document root <none> and password file <none>
I20260926 03:04:11.536759 18130 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:11.536912 18130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:11.537308 18130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:11.540117 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "051477b77d994e5fa6a864a779e62b2b"
format_stamp: "Formatted at 2026-09-26 03:04:11 on dist-test-slave-fccw"
server_key: "c8fc7b9107e723014f34ab12a1596c3a"
server_key_iv: "a587a954b81c93b877a137e943ef68a2"
server_key_version: "encryptionkey@0"
I20260926 03:04:11.540927 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "051477b77d994e5fa6a864a779e62b2b"
format_stamp: "Formatted at 2026-09-26 03:04:11 on dist-test-slave-fccw"
server_key: "c8fc7b9107e723014f34ab12a1596c3a"
server_key_iv: "a587a954b81c93b877a137e943ef68a2"
server_key_version: "encryptionkey@0"
I20260926 03:04:11.545938 18130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 03:04:11.549212 18144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:11.550470 18130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:04:11.550647 18130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "051477b77d994e5fa6a864a779e62b2b"
format_stamp: "Formatted at 2026-09-26 03:04:11 on dist-test-slave-fccw"
server_key: "c8fc7b9107e723014f34ab12a1596c3a"
server_key_iv: "a587a954b81c93b877a137e943ef68a2"
server_key_version: "encryptionkey@0"
I20260926 03:04:11.550881 18130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:11.580545 18130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:11.581444 18130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:11.581725 18130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:11.582720 18130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:11.584182 18130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:11.584304 18130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:11.584445 18130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:11.584532 18130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:11.617734 18130 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43899
I20260926 03:04:11.617826 18256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43899 every 8 connection(s)
I20260926 03:04:11.619073 18130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:04:11.625793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18130
I20260926 03:04:11.625976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:04:11.626394 388 external_mini_cluster.cc:1428] Setting key e2d651bb2dcd092b651e81388b734610
I20260926 03:04:11.628957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39547
--builtin_ntp_servers=127.0.97.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:04:11.631850 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:11.621719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49399 (local address 127.0.97.62:39547)
0926 03:04:11.622191 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:11.622195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:11.623273 (+ 1078us) server_negotiation.cc:407] Connection header received
0926 03:04:11.625144 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:11.625150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:11.625239 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:11.625364 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:11.626512 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:11.627421 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:11.629138 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:11.629556 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:11.629693 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:11.631044 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:11.631048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:11.631052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:11.631068 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:11.631108 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:11.631119 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:11.631212 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:11.631430 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 03:04:11.631526 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":270,"threads_started":1}
I20260926 03:04:11.632906 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:11.622232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39547 (local address 127.0.97.1:49399)
0926 03:04:11.623116 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:11.623151 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:11.624805 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:11.625608 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:11.625616 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:11.625704 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:11.626350 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:11.626367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:11.627562 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:11.627572 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:11.628957 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:11.628970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:11.629777 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:11.629784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:11.630018 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:11.630718 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:11.630721 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:11.630760 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:11.630835 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:11.631213 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:11.631226 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:11.631236 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:04:11.631418 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 03:04:11.631808 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":278,"threads_started":1}
I20260926 03:04:11.634426 18257 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39547
I20260926 03:04:11.634745 18257 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:11.635644 18257 heartbeater.cc:511] Master 127.0.97.62:39547 requested a full tablet report, sending...
I20260926 03:04:11.637970 18071 ts_manager.cc:194] Registered new tserver with Master: 051477b77d994e5fa6a864a779e62b2b (127.0.97.1:43899)
I20260926 03:04:11.640237 18071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:49399
W20260926 03:04:11.954378 18261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:11.954698 18261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:11.954861 18261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:11.961761 18261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:11.961899 18261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:11.962076 18261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:04:11.970707 18261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:11.971932 18261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:11.973768 18261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:11.986738 18269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:11.986686 18267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:11.987444 18266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:11.988155 18261 server_base.cc:1034] running on GCE node
I20260926 03:04:11.988715 18261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:11.989612 18261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:11.990811 18261 hybrid_clock.cc:648] HybridClock initialized: now 1790391851990776 us; error 26 us; skew 500 ppm
I20260926 03:04:11.991118 18261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:11.993419 18261 webserver.cc:460] Webserver started at http://127.0.97.2:44097/ using document root <none> and password file <none>
I20260926 03:04:11.994086 18261 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:11.994195 18261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:11.994534 18261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:11.997308 18261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "d132e004486d4f2c8140950132f64588"
format_stamp: "Formatted at 2026-09-26 03:04:11 on dist-test-slave-fccw"
server_key: "5334b9ab4290889506306228f43fc132"
server_key_iv: "1656acab0c2094b46e4fed275cb83223"
server_key_version: "encryptionkey@0"
I20260926 03:04:11.998037 18261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d132e004486d4f2c8140950132f64588"
format_stamp: "Formatted at 2026-09-26 03:04:11 on dist-test-slave-fccw"
server_key: "5334b9ab4290889506306228f43fc132"
server_key_iv: "1656acab0c2094b46e4fed275cb83223"
server_key_version: "encryptionkey@0"
I20260926 03:04:12.003103 18261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:04:12.006434 18275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:12.007619 18261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:04:12.007797 18261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "d132e004486d4f2c8140950132f64588"
format_stamp: "Formatted at 2026-09-26 03:04:11 on dist-test-slave-fccw"
server_key: "5334b9ab4290889506306228f43fc132"
server_key_iv: "1656acab0c2094b46e4fed275cb83223"
server_key_version: "encryptionkey@0"
I20260926 03:04:12.008004 18261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:12.028276 18261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:12.029155 18261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:12.029426 18261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:12.030361 18261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:12.031766 18261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:12.031886 18261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:12.032009 18261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:12.032101 18261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:12.064612 18261 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33339
I20260926 03:04:12.064666 18387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33339 every 8 connection(s)
I20260926 03:04:12.066185 18261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:04:12.074674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18261
I20260926 03:04:12.074856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:04:12.075398 388 external_mini_cluster.cc:1428] Setting key 791e938168baa2bf2c1a4802de15eb18
I20260926 03:04:12.078373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39547
--builtin_ntp_servers=127.0.97.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:04:12.079293 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.068872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51499 (local address 127.0.97.62:39547)
0926 03:04:12.069110 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.069114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.070574 (+ 1460us) server_negotiation.cc:407] Connection header received
0926 03:04:12.072345 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.072353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.072455 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.072575 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:12.073646 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.074617 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.076459 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.076855 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.076990 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.078596 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:12.078604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:12.078608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:12.078627 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:12.078662 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:12.078678 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:12.078762 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.079024 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.079095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:04:12.080825 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.069463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39547 (local address 127.0.97.2:51499)
0926 03:04:12.070423 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.070457 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:12.072064 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:12.072790 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:12.072798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:12.072890 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:12.073489 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.073505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.074827 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.074832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.076315 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.076326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.077127 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.077138 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.077429 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.078211 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:12.078216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:12.078273 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:12.078383 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:12.078793 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:12.078805 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:12.078816 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:04:12.078993 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 03:04:12.079494 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":316,"threads_started":1}
I20260926 03:04:12.082201 18388 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39547
I20260926 03:04:12.082507 18388 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:12.083209 18388 heartbeater.cc:511] Master 127.0.97.62:39547 requested a full tablet report, sending...
I20260926 03:04:12.084553 18071 ts_manager.cc:194] Registered new tserver with Master: d132e004486d4f2c8140950132f64588 (127.0.97.2:33339)
I20260926 03:04:12.085675 18071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51499
W20260926 03:04:12.402683 18391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:12.402953 18391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:12.403071 18391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:12.410297 18391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:12.410386 18391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:12.410543 18391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:04:12.419577 18391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:12.420835 18391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:12.422613 18391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:12.435674 18397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:12.435751 18396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:12.435783 18399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:12.436311 18391 server_base.cc:1034] running on GCE node
I20260926 03:04:12.436913 18391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:12.437852 18391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:12.439080 18391 hybrid_clock.cc:648] HybridClock initialized: now 1790391852439046 us; error 47 us; skew 500 ppm
I20260926 03:04:12.439389 18391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:12.441797 18391 webserver.cc:460] Webserver started at http://127.0.97.3:37097/ using document root <none> and password file <none>
I20260926 03:04:12.442468 18391 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:12.442597 18391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:12.442917 18391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:12.445680 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "4b8c5bcd33ca48e79a3abd689c422488"
format_stamp: "Formatted at 2026-09-26 03:04:12 on dist-test-slave-fccw"
server_key: "1e0d9579dc96c910e006971948be5508"
server_key_iv: "b7638cdc2fc0dfa2278772b79e4361de"
server_key_version: "encryptionkey@0"
I20260926 03:04:12.446478 18391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4b8c5bcd33ca48e79a3abd689c422488"
format_stamp: "Formatted at 2026-09-26 03:04:12 on dist-test-slave-fccw"
server_key: "1e0d9579dc96c910e006971948be5508"
server_key_iv: "b7638cdc2fc0dfa2278772b79e4361de"
server_key_version: "encryptionkey@0"
I20260926 03:04:12.451274 18391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 03:04:12.454563 18405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:12.455930 18391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:04:12.456099 18391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "4b8c5bcd33ca48e79a3abd689c422488"
format_stamp: "Formatted at 2026-09-26 03:04:12 on dist-test-slave-fccw"
server_key: "1e0d9579dc96c910e006971948be5508"
server_key_iv: "b7638cdc2fc0dfa2278772b79e4361de"
server_key_version: "encryptionkey@0"
I20260926 03:04:12.456306 18391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:12.473263 18391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:12.474056 18391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:12.474334 18391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:12.475334 18391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:12.476914 18391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:12.477042 18391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:12.477173 18391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:12.477265 18391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:12.509161 18391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46329
I20260926 03:04:12.509207 18517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46329 every 8 connection(s)
I20260926 03:04:12.510492 18391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:04:12.512524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18391
I20260926 03:04:12.512728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:04:12.513321 388 external_mini_cluster.cc:1428] Setting key 3427bf53f6bce33aca2cbd3362947f22
I20260926 03:04:12.524329 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.513619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60561 (local address 127.0.97.62:39547)
0926 03:04:12.513838 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.513845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.515510 (+ 1665us) server_negotiation.cc:407] Connection header received
0926 03:04:12.517702 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.517708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.517806 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.517920 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:12.519040 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.519933 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.521558 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.521984 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.522113 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.523508 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:12.523514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:12.523520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:12.523545 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:12.523574 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:12.523586 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:12.523681 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.523902 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.524001 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 03:04:12.525417 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.514175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39547 (local address 127.0.97.3:60561)
0926 03:04:12.515361 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.515406 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:12.517427 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:12.518177 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:12.518186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:12.518283 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:12.518876 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.518892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.520085 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.520090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.521408 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.521420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.522224 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.522231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.522460 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.523176 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:12.523179 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:12.523221 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:12.523297 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:12.523681 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:12.523695 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:12.523705 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:04:12.523892 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 03:04:12.524295 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":397,"threads_started":1}
I20260926 03:04:12.526741 18518 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39547
I20260926 03:04:12.527122 18518 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:12.528034 18518 heartbeater.cc:511] Master 127.0.97.62:39547 requested a full tablet report, sending...
I20260926 03:04:12.529487 18071 ts_manager.cc:194] Registered new tserver with Master: 4b8c5bcd33ca48e79a3abd689c422488 (127.0.97.3:46329)
I20260926 03:04:12.530254 18071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60561
I20260926 03:04:12.540827 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:04:12.550145 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.544699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.97.62:39547)
0926 03:04:12.544953 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.544960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.545256 (+ 296us) server_negotiation.cc:407] Connection header received
0926 03:04:12.545388 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.545392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.545465 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.545543 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:12.546004 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.546687 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.547921 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.548301 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.548394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.549464 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:12.549468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:12.549471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:12.549489 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:12.549518 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:12.549530 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:12.549627 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.549765 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.549851 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:04:12.553539 18071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:04:12.560397 18071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:04:12.583667 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.572567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.0.97.1:43899)
0926 03:04:12.573490 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.573498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.574512 (+ 1014us) server_negotiation.cc:407] Connection header received
0926 03:04:12.574778 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.574785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.574947 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.575100 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:12.576115 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.577814 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.580585 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.582072 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.582197 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.582470 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.583120 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.583292 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":688,"threads_started":1}
I20260926 03:04:12.584463 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.571955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52242)
0926 03:04:12.574406 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.574424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:12.574608 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:12.575421 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:12.575430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:12.575462 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:12.575877 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.575890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.577987 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.577991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.580332 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.580345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.582639 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.582646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.582847 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.582905 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 03:04:12.583944 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0926 03:04:12.584129 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2315,"thread_start_us":1714,"threads_started":1}
I20260926 03:04:12.586571 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.572102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.0.97.2:33339)
0926 03:04:12.573091 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.573103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.574228 (+ 1125us) server_negotiation.cc:407] Connection header received
0926 03:04:12.574640 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.574668 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.574934 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.575101 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:12.577074 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.578524 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.582353 (+ 3829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.583781 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.583899 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.584122 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.585780 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.586068 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":643,"threads_started":1}
I20260926 03:04:12.587651 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.571902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33339 (local address 127.0.0.1:33600)
0926 03:04:12.574058 (+ 2156us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.574113 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:12.574395 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:12.576406 (+ 2011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:12.576410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:12.576434 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:12.576847 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.576858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.578694 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.578699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.582154 (+ 3455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.582167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.585246 (+ 3079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.585257 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.585524 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.585575 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 03:04:12.586993 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0926 03:04:12.587199 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1879,"thread_start_us":2207,"threads_started":1}
I20260926 03:04:12.589530 18192 tablet_service.cc:1467] Processing CreateTablet for tablet e039d8168990446b847f05663484aa19 (DEFAULT_TABLE table=test-table [id=d5963872b8cb4a22b63f6b8ed12ab009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:12.591041 18192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e039d8168990446b847f05663484aa19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:12.591225 18323 tablet_service.cc:1467] Processing CreateTablet for tablet e039d8168990446b847f05663484aa19 (DEFAULT_TABLE table=test-table [id=d5963872b8cb4a22b63f6b8ed12ab009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:12.592519 18323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e039d8168990446b847f05663484aa19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:12.594558 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.574594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46329 (local address 127.0.0.1:48888)
0926 03:04:12.584803 (+ 10209us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.584821 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:12.584925 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:12.589367 (+ 4442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:12.589372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:12.589413 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:12.589574 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.589586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.591311 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.591314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.592495 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.592506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.593790 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.593796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.593988 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.594031 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:04:12.594129 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:04:12.594262 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10102,"thread_start_us":13542,"threads_started":1}
I20260926 03:04:12.594825 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.574937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.0.97.3:46329)
0926 03:04:12.588712 (+ 13775us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.588721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.588754 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:04:12.588861 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.588867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.589145 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.589748 (+ 603us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:12.590045 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.591091 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.592651 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.593623 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.593766 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.594041 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.594172 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.594318 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13555,"thread_start_us":13663,"threads_started":1}
I20260926 03:04:12.598405 18453 tablet_service.cc:1467] Processing CreateTablet for tablet e039d8168990446b847f05663484aa19 (DEFAULT_TABLE table=test-table [id=d5963872b8cb4a22b63f6b8ed12ab009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:12.599778 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e039d8168990446b847f05663484aa19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:12.605026 18536 tablet_bootstrap.cc:492] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b: Bootstrap starting.
I20260926 03:04:12.606688 18537 tablet_bootstrap.cc:492] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Bootstrap starting.
I20260926 03:04:12.608829 18536 tablet_bootstrap.cc:654] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:12.610111 18536 log.cc:824] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:12.610165 18537 tablet_bootstrap.cc:654] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:12.611346 18537 log.cc:824] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:12.612967 18540 tablet_bootstrap.cc:492] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Bootstrap starting.
I20260926 03:04:12.614122 18536 tablet_bootstrap.cc:492] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b: No bootstrap required, opened a new log
I20260926 03:04:12.614400 18536 ts_tablet_manager.cc:1397] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260926 03:04:12.614640 18537 tablet_bootstrap.cc:492] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: No bootstrap required, opened a new log
I20260926 03:04:12.614924 18537 ts_tablet_manager.cc:1397] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 03:04:12.616571 18540 tablet_bootstrap.cc:654] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:12.617723 18540 log.cc:824] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:12.620414 18540 tablet_bootstrap.cc:492] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: No bootstrap required, opened a new log
I20260926 03:04:12.620724 18540 ts_tablet_manager.cc:1397] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260926 03:04:12.626061 18537 raft_consensus.cc:348] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.626577 18537 raft_consensus.cc:374] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:12.626701 18537 raft_consensus.cc:729] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d132e004486d4f2c8140950132f64588, State: Initialized, Role: FOLLOWER
I20260926 03:04:12.626693 18536 raft_consensus.cc:348] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.627233 18536 raft_consensus.cc:374] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:12.627395 18536 raft_consensus.cc:729] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051477b77d994e5fa6a864a779e62b2b, State: Initialized, Role: FOLLOWER
I20260926 03:04:12.627454 18537 consensus_queue.cc:260] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.628093 18536 consensus_queue.cc:260] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.631964 18540 raft_consensus.cc:348] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.632429 18540 raft_consensus.cc:374] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:12.632573 18540 raft_consensus.cc:729] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8c5bcd33ca48e79a3abd689c422488, State: Initialized, Role: FOLLOWER
I20260926 03:04:12.633358 18536 ts_tablet_manager.cc:1428] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260926 03:04:12.633854 18257 heartbeater.cc:503] Master 127.0.97.62:39547 was elected leader, sending a full tablet report...
I20260926 03:04:12.639740 18540 consensus_queue.cc:260] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.640264 18544 raft_consensus.cc:482] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:04:12.640966 18518 heartbeater.cc:503] Master 127.0.97.62:39547 was elected leader, sending a full tablet report...
I20260926 03:04:12.641510 18388 heartbeater.cc:503] Master 127.0.97.62:39547 was elected leader, sending a full tablet report...
I20260926 03:04:12.642206 18537 ts_tablet_manager.cc:1428] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Time spent starting tablet: real 0.027s user 0.011s sys 0.014s
I20260926 03:04:12.642251 18540 ts_tablet_manager.cc:1428] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Time spent starting tablet: real 0.021s user 0.008s sys 0.011s
I20260926 03:04:12.640596 18544 raft_consensus.cc:504] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.646664 18544 leader_election.cc:288] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d132e004486d4f2c8140950132f64588 (127.0.97.2:33339), 051477b77d994e5fa6a864a779e62b2b (127.0.97.1:43899)
I20260926 03:04:12.652935 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.646200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56313 (local address 127.0.97.1:43899)
0926 03:04:12.646440 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.646446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.646463 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:04:12.646522 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.646527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.646611 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.646703 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:12.647358 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.648428 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.650995 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.652000 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.652093 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.652154 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.652553 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.652679 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:04:12.652945 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.645864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58065 (local address 127.0.97.2:33339)
0926 03:04:12.646033 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.646046 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.646818 (+ 772us) server_negotiation.cc:407] Connection header received
0926 03:04:12.647084 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.647089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.647166 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.647212 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:12.648000 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.648753 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.651219 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.652106 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.652228 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.652347 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.652602 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.652718 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:04:12.652954 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.645783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.97.3:56313)
0926 03:04:12.645932 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.645956 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:12.646066 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:12.646930 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:12.646937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:12.646968 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:12.647170 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.647181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.648568 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.648574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.650791 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.650818 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.652144 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.652153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.652362 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.652401 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:04:12.652496 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:04:12.652661 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":122}
I20260926 03:04:12.653112 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.645783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33339 (local address 127.0.97.3:58065)
0926 03:04:12.646723 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.646748 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:12.646931 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:12.647426 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:12.647430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:12.647455 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:12.647851 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.647862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.648947 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.648953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.651043 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:12.651055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.652296 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.652304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:12.652432 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.652454 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 03:04:12.652562 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:04:12.652681 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":217,"thread_start_us":406,"threads_started":1}
I20260926 03:04:12.654271 18343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e039d8168990446b847f05663484aa19" candidate_uuid: "4b8c5bcd33ca48e79a3abd689c422488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d132e004486d4f2c8140950132f64588" is_pre_election: true
I20260926 03:04:12.654402 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e039d8168990446b847f05663484aa19" candidate_uuid: "4b8c5bcd33ca48e79a3abd689c422488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051477b77d994e5fa6a864a779e62b2b" is_pre_election: true
I20260926 03:04:12.654767 18343 raft_consensus.cc:2454] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b8c5bcd33ca48e79a3abd689c422488 in term 0.
I20260926 03:04:12.654875 18212 raft_consensus.cc:2454] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b8c5bcd33ca48e79a3abd689c422488 in term 0.
I20260926 03:04:12.655332 18407 leader_election.cc:302] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8c5bcd33ca48e79a3abd689c422488, d132e004486d4f2c8140950132f64588; no voters:
I20260926 03:04:12.655730 18544 raft_consensus.cc:2781] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:04:12.655874 18544 raft_consensus.cc:482] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:04:12.655989 18544 raft_consensus.cc:3037] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:12.658120 18544 raft_consensus.cc:504] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.658667 18544 leader_election.cc:288] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [CANDIDATE]: Term 1 election: Requested vote from peers d132e004486d4f2c8140950132f64588 (127.0.97.2:33339), 051477b77d994e5fa6a864a779e62b2b (127.0.97.1:43899)
I20260926 03:04:12.659173 18343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e039d8168990446b847f05663484aa19" candidate_uuid: "4b8c5bcd33ca48e79a3abd689c422488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d132e004486d4f2c8140950132f64588"
I20260926 03:04:12.659226 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e039d8168990446b847f05663484aa19" candidate_uuid: "4b8c5bcd33ca48e79a3abd689c422488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051477b77d994e5fa6a864a779e62b2b"
I20260926 03:04:12.659412 18212 raft_consensus.cc:3037] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:12.659467 18343 raft_consensus.cc:3037] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:12.661826 18212 raft_consensus.cc:2454] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b8c5bcd33ca48e79a3abd689c422488 in term 1.
I20260926 03:04:12.661931 18343 raft_consensus.cc:2454] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b8c5bcd33ca48e79a3abd689c422488 in term 1.
I20260926 03:04:12.662256 18409 leader_election.cc:302] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051477b77d994e5fa6a864a779e62b2b, 4b8c5bcd33ca48e79a3abd689c422488; no voters:
I20260926 03:04:12.662618 18544 raft_consensus.cc:2781] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:12.662768 18544 raft_consensus.cc:686] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 1 LEADER]: Becoming Leader. State: Replica: 4b8c5bcd33ca48e79a3abd689c422488, State: Running, Role: LEADER
I20260926 03:04:12.663216 18544 consensus_queue.cc:237] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } }
I20260926 03:04:12.667282 18070 catalog_manager.cc:5495] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b8c5bcd33ca48e79a3abd689c422488 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4b8c5bcd33ca48e79a3abd689c422488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b8c5bcd33ca48e79a3abd689c422488" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46329 } health_report { overall_health: HEALTHY } } }
I20260926 03:04:12.681481 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.676809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.0.97.3:46329)
0926 03:04:12.676963 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.676967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.676985 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:04:12.677208 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.677211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.677284 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.677395 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:04:12.677660 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.678338 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.679620 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.679991 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.680085 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.680481 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:04:12.680947 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:04:12.681126 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.681251 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.681307 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:04:12.688972 18212 raft_consensus.cc:1261] T e039d8168990446b847f05663484aa19 P 051477b77d994e5fa6a864a779e62b2b [term 1 FOLLOWER]: Refusing update from remote peer 4b8c5bcd33ca48e79a3abd689c422488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:12.688984 18343 raft_consensus.cc:1261] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 1 FOLLOWER]: Refusing update from remote peer 4b8c5bcd33ca48e79a3abd689c422488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:12.689882 18545 consensus_queue.cc:1035] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051477b77d994e5fa6a864a779e62b2b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:12.690272 18544 consensus_queue.cc:1035] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d132e004486d4f2c8140950132f64588" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:12.701261 18549 mvcc.cc:205] Tried to move back new op lower bound from 7333445028604653568 to 7333445028512296960. Current Snapshot: MvccSnapshot[applied={T|T < 7333445028604653568}]
I20260926 03:04:12.721892 18071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:12.722177 18071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:12.725438 18071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b572997e7e84c6fbe5360fe810475cc: Sending DeleteTablet for 3 replicas of tablet e039d8168990446b847f05663484aa19
I20260926 03:04:12.726536 18323 tablet_service.cc:1514] Processing DeleteTablet for tablet e039d8168990446b847f05663484aa19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:12 UTC) from {username='slave'} at 127.0.0.1:33600
I20260926 03:04:12.726816 18192 tablet_service.cc:1514] Processing DeleteTablet for tablet e039d8168990446b847f05663484aa19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:12 UTC) from {username='slave'} at 127.0.0.1:52242
I20260926 03:04:12.727541 18559 tablet_replica.cc:330] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: stopping tablet replica
I20260926 03:04:12.727896 18453 tablet_service.cc:1514] Processing DeleteTablet for tablet e039d8168990446b847f05663484aa19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:12 UTC) from {username='slave'} at 127.0.0.1:48888
I20260926 03:04:12.728080 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18130
I20260926 03:04:12.728015 18559 raft_consensus.cc:2229] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:12.728663 18559 raft_consensus.cc:2258] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:12.732975 18559 ts_tablet_manager.cc:1905] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:12.739560 18533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.729213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.97.1:43899)
0926 03:04:12.729380 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.729394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.730797 (+ 1403us) server_negotiation.cc:407] Connection header received
0926 03:04:12.731112 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.731124 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.731273 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.731471 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:12.732240 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.733421 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.735405 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.736084 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.736178 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.737876 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:12.737888 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:12.737904 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:12.738660 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:12.738736 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:12.738762 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:12.738848 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.739094 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.739201 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:04:12.740235 18559 ts_tablet_manager.cc:1918] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:12.740579 18559 log.cc:1196] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/e039d8168990446b847f05663484aa19
I20260926 03:04:12.740948 18252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:12.741109 18559 ts_tablet_manager.cc:1939] T e039d8168990446b847f05663484aa19 P d132e004486d4f2c8140950132f64588: Deleting consensus metadata
I20260926 03:04:12.743112 18056 catalog_manager.cc:4841] TS d132e004486d4f2c8140950132f64588 (127.0.97.2:33339): tablet e039d8168990446b847f05663484aa19 (table test-table [id=d5963872b8cb4a22b63f6b8ed12ab009]) successfully deleted
I20260926 03:04:12.743811 18561 tablet_replica.cc:330] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: stopping tablet replica
I20260926 03:04:12.744184 18561 raft_consensus.cc:2229] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:04:12.744756 18561 raft_consensus.cc:2258] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:12.746845 18561 ts_tablet_manager.cc:1905] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:12.754705 18561 ts_tablet_manager.cc:1918] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:12.754945 18561 log.cc:1196] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/e039d8168990446b847f05663484aa19
I20260926 03:04:12.755607 18561 ts_tablet_manager.cc:1939] T e039d8168990446b847f05663484aa19 P 4b8c5bcd33ca48e79a3abd689c422488: Deleting consensus metadata
I20260926 03:04:12.757690 18057 catalog_manager.cc:4841] TS 4b8c5bcd33ca48e79a3abd689c422488 (127.0.97.3:46329): tablet e039d8168990446b847f05663484aa19 (table test-table [id=d5963872b8cb4a22b63f6b8ed12ab009]) successfully deleted
I20260926 03:04:12.834321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18130
W20260926 03:04:12.855484 18058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43899 (error 108)
I20260926 03:04:12.856254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18261
I20260926 03:04:12.857726 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.857147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52264)
0926 03:04:12.857427 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.857520 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260926 03:04:12.857995 18058 catalog_manager.cc:4584] TS 051477b77d994e5fa6a864a779e62b2b (127.0.97.1:43899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e039d8168990446b847f05663484aa19: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
I20260926 03:04:12.863823 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.857257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.0.97.2:33339)
0926 03:04:12.857449 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:12.857455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:12.857475 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:04:12.857640 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:12.857648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:12.857797 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:12.857913 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:12.858531 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.859256 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.860670 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:12.861220 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:12.861353 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:12.862496 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:12.862506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:12.862520 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:12.863086 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:12.863132 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:12.863150 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:12.863286 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:12.863420 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 03:04:12.863523 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:04:12.864471 18383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:12.911497 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.911047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52274)
0926 03:04:12.911251 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.911343 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:04:12.912644 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.912283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52288)
0926 03:04:12.912440 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.912520 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 03:04:12.973196 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.972689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52300)
0926 03:04:12.972918 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.973039 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:04:12.974247 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:12.973905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52304)
0926 03:04:12.974042 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:12.974134 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 03:04:13.012869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18261
I20260926 03:04:13.029917 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18391
I20260926 03:04:13.037968 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:13.030716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.0.97.3:46329)
0926 03:04:13.030921 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:13.030928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:13.031068 (+ 140us) server_negotiation.cc:407] Connection header received
0926 03:04:13.031245 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:13.031249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:13.031324 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:13.031415 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:13.031880 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:13.032685 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:13.034567 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:13.035029 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:13.035147 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:13.036411 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:13.036429 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:13.036444 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:13.037185 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:13.037235 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:13.037255 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:13.037333 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:13.037524 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 03:04:13.037594 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 03:04:13.038698 18513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:13.055063 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:13.054441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52314)
0926 03:04:13.054708 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:13.054840 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 03:04:13.056375 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:13.055848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52320)
0926 03:04:13.056081 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:13.056183 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 03:04:13.164124 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18391
I20260926 03:04:13.184365 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18040
I20260926 03:04:13.185529 18101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:13.293854 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:13.292909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43899 (local address 127.0.0.1:52324)
0926 03:04:13.293359 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:13.293486 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 03:04:13.293984 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18040
2026-09-26T03:04:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2731 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:13 dist-test-slave-fccw krb5kdc[18570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:13 dist-test-slave-fccw krb5kdc[18570](info): set up 2 sockets
Sep 26 03:04:13 dist-test-slave-fccw krb5kdc[18570](info): commencing operation
krb5kdc: starting...
W20260926 03:04:14.405177 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:14 dist-test-slave-fccw krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:14.640316 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33277
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:14.961431 18583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:14.961758 18583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:14.961925 18583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:14.968911 18583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:14.969074 18583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:14.969173 18583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:14.969269 18583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:14.969359 18583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:14.978812 18583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:14.980173 18583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:14.981688 18583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:14.993448 18591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:14.993852 18588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:14.993885 18589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:14.994730 18583 server_base.cc:1034] running on GCE node
I20260926 03:04:14.995425 18583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:14.996722 18583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:14.997943 18583 hybrid_clock.cc:648] HybridClock initialized: now 1790391854997900 us; error 53 us; skew 500 ppm
I20260926 03:04:14.998286 18583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:14 dist-test-slave-fccw krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:15.001439 18583 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:04:15.003002 18583 webserver.cc:460] Webserver started at http://127.0.97.62:42183/ using document root <none> and password file <none>
I20260926 03:04:15.003683 18583 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:15.003839 18583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:15.004148 18583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:15.007130 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "240a413ed0ab481884b8147d2991ead6"
format_stamp: "Formatted at 2026-09-26 03:04:15 on dist-test-slave-fccw"
server_key: "0c42e0054c419b0a97e3cad03e294789"
server_key_iv: "8f733b64c86545930dff73b8666b097a"
server_key_version: "encryptionkey@0"
I20260926 03:04:15.007905 18583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "240a413ed0ab481884b8147d2991ead6"
format_stamp: "Formatted at 2026-09-26 03:04:15 on dist-test-slave-fccw"
server_key: "0c42e0054c419b0a97e3cad03e294789"
server_key_iv: "8f733b64c86545930dff73b8666b097a"
server_key_version: "encryptionkey@0"
I20260926 03:04:15.012842 18583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 03:04:15.016196 18598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:15.017516 18583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:04:15.017704 18583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "240a413ed0ab481884b8147d2991ead6"
format_stamp: "Formatted at 2026-09-26 03:04:15 on dist-test-slave-fccw"
server_key: "0c42e0054c419b0a97e3cad03e294789"
server_key_iv: "8f733b64c86545930dff73b8666b097a"
server_key_version: "encryptionkey@0"
I20260926 03:04:15.017935 18583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:15.056295 18583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:15.060250 18583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:15.060542 18583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:15.079051 18583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33277
I20260926 03:04:15.079077 18649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33277 every 8 connection(s)
I20260926 03:04:15.080480 18583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:15.082670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18583
I20260926 03:04:15.082856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:15.083433 388 external_mini_cluster.cc:1428] Setting key 2668ca2f666bb120bdc9e0fa14036da3
I20260926 03:04:15.085093 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:04:15 dist-test-slave-fccw krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:15.099388 18650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6: Bootstrap starting.
I20260926 03:04:15.103513 18650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:15.104550 18650 log.cc:824] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:15.107311 18650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6: No bootstrap required, opened a new log
I20260926 03:04:15.108285 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:15.085329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.0.97.62:33277)
0926 03:04:15.086070 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:15.086081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:15.086120 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:04:15.087649 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:15.087669 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:15.088025 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:15.088351 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:15.090101 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:15.091372 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:15.093328 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:15.094559 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:15.094781 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:15.100075 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:15.100083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:15.100096 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:15.100136 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:15.104050 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:15.104821 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:15.104829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:15.104835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:15.104989 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:15.105427 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:15.105432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:15.105436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:15.105955 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:15.106266 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:15.106715 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 03:04:15.107291 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":334,"threads_started":1}
I20260926 03:04:15.114498 18650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a413ed0ab481884b8147d2991ead6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33277 } }
I20260926 03:04:15.114866 18650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:15.114989 18650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240a413ed0ab481884b8147d2991ead6, State: Initialized, Role: FOLLOWER
I20260926 03:04:15.115507 18650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a413ed0ab481884b8147d2991ead6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33277 } }
I20260926 03:04:15.115738 18650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:15.115877 18650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:15.116057 18650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:15.118244 18650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a413ed0ab481884b8147d2991ead6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33277 } }
I20260926 03:04:15.118695 18650 leader_election.cc:302] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 240a413ed0ab481884b8147d2991ead6; no voters:
I20260926 03:04:15.119237 18650 leader_election.cc:288] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:15.119364 18655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:15.120070 18655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [term 1 LEADER]: Becoming Leader. State: Replica: 240a413ed0ab481884b8147d2991ead6, State: Running, Role: LEADER
I20260926 03:04:15.120605 18655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a413ed0ab481884b8147d2991ead6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33277 } }
I20260926 03:04:15.121372 18650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:15.123407 18656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "240a413ed0ab481884b8147d2991ead6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a413ed0ab481884b8147d2991ead6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33277 } } }
I20260926 03:04:15.123731 18656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:15.124549 18661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:15.124434 18657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240a413ed0ab481884b8147d2991ead6. Latest consensus state: current_term: 1 leader_uuid: "240a413ed0ab481884b8147d2991ead6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240a413ed0ab481884b8147d2991ead6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33277 } } }
I20260926 03:04:15.124769 18657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:15.129829 18661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:15.138049 18661 catalog_manager.cc:1347] Generated new cluster ID: ed5730b46c5643f38dbf153738ad1ed1
I20260926 03:04:15.138167 18661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:15.146680 18661 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:15.147939 18661 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:15.160387 18661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6: Generated new TSK 0
I20260926 03:04:15.161270 18661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:04:15.215411 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33277
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:15.546180 18676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:15.546448 18676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:15.546625 18676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:15.553685 18676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:15.553814 18676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:15.554008 18676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:04:15.563344 18676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:15.564637 18676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:15.565963 18676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:15.578962 18681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:15.579012 18684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:15.579056 18682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:15.579823 18676 server_base.cc:1034] running on GCE node
I20260926 03:04:15.580480 18676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:15.581496 18676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:15.582727 18676 hybrid_clock.cc:648] HybridClock initialized: now 1790391855582684 us; error 34 us; skew 500 ppm
I20260926 03:04:15.583058 18676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:15 dist-test-slave-fccw krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:15.586313 18676 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:04:15.587713 18676 webserver.cc:460] Webserver started at http://127.0.97.1:40769/ using document root <none> and password file <none>
I20260926 03:04:15.588424 18676 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:15.588580 18676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:15.588896 18676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:15.591874 18676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "c74f28ff05b346cab02faeff58e5cb37"
format_stamp: "Formatted at 2026-09-26 03:04:15 on dist-test-slave-fccw"
server_key: "acfb435074443dce4700963a8b1f1d46"
server_key_iv: "9b08c064621978624d648c4f26d10a37"
server_key_version: "encryptionkey@0"
I20260926 03:04:15.592680 18676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c74f28ff05b346cab02faeff58e5cb37"
format_stamp: "Formatted at 2026-09-26 03:04:15 on dist-test-slave-fccw"
server_key: "acfb435074443dce4700963a8b1f1d46"
server_key_iv: "9b08c064621978624d648c4f26d10a37"
server_key_version: "encryptionkey@0"
I20260926 03:04:15.597689 18676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:04:15.600955 18691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:15.602249 18676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:15.602425 18676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "c74f28ff05b346cab02faeff58e5cb37"
format_stamp: "Formatted at 2026-09-26 03:04:15 on dist-test-slave-fccw"
server_key: "acfb435074443dce4700963a8b1f1d46"
server_key_iv: "9b08c064621978624d648c4f26d10a37"
server_key_version: "encryptionkey@0"
I20260926 03:04:15.602622 18676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:15.619941 18676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:15.624462 18676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:15.624747 18676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:15.625747 18676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:15.627105 18676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:15.627226 18676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:15.627372 18676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:15.627445 18676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:15.660524 18676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45275
I20260926 03:04:15.660665 18803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45275 every 8 connection(s)
I20260926 03:04:15.661830 18676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:04:15.669448 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18676
I20260926 03:04:15.669632 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:04:15.670042 388 external_mini_cluster.cc:1428] Setting key 86d1697a5e6e17e46d2abc10a135376c
Sep 26 03:04:15 dist-test-slave-fccw krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:15.681908 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:15.664508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49785 (local address 127.0.97.62:33277)
0926 03:04:15.664961 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:15.664965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:15.665855 (+ 890us) server_negotiation.cc:407] Connection header received
0926 03:04:15.667675 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:15.667682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:15.667770 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:15.667896 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:15.669388 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:15.670260 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:15.672084 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:15.672542 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:15.672683 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:15.676294 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:15.676299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:15.676302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:15.676332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:15.678879 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:15.679608 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:15.679616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:15.679624 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:15.679736 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:15.680174 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:15.680179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:15.680182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:15.680434 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:15.680596 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:15.681326 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 03:04:15.681550 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":236,"threads_started":1}
I20260926 03:04:15.683043 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:15.664973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33277 (local address 127.0.97.1:49785)
0926 03:04:15.665702 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:15.665744 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:15.667412 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:15.668125 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:15.668133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:15.668620 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:15.669223 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:15.669241 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:15.670433 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:15.670436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:15.671930 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:15.671941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:15.672791 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:15.672800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:15.673090 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:15.673771 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:15.673797 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:15.676074 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:15.679030 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:15.679039 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:15.679055 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:15.679481 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:15.679845 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:15.679851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:15.679855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:15.680058 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:15.680558 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:15.680567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:15.680903 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 03:04:15.681222 (+ 319us) client_negotiation.cc:240] Negotiation successful
0926 03:04:15.681725 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":269,"threads_started":1}
I20260926 03:04:15.684645 18804 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33277
I20260926 03:04:15.684947 18804 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:15.685801 18804 heartbeater.cc:511] Master 127.0.97.62:33277 requested a full tablet report, sending...
I20260926 03:04:15.688184 18615 ts_manager.cc:194] Registered new tserver with Master: c74f28ff05b346cab02faeff58e5cb37 (127.0.97.1:45275)
I20260926 03:04:15.690433 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49785
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:04:15.723610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33277
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:16.056293 18810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:16.056640 18810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:16.056825 18810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:16.064316 18810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:16.064467 18810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:16.064637 18810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:04:16.074790 18810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:16.076133 18810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:16.077540 18810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:16.091426 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:16.091404 18818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:16.091828 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:16.092336 18810 server_base.cc:1034] running on GCE node
I20260926 03:04:16.093044 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:16.094076 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:16.095299 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790391856095250 us; error 42 us; skew 500 ppm
I20260926 03:04:16.095629 18810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:16 dist-test-slave-fccw krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:16.098696 18810 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:04:16.100196 18810 webserver.cc:460] Webserver started at http://127.0.97.2:35853/ using document root <none> and password file <none>
I20260926 03:04:16.100977 18810 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:16.101197 18810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:16.101516 18810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:16.104420 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "63996c50a44f415f883a358f486327ee"
format_stamp: "Formatted at 2026-09-26 03:04:16 on dist-test-slave-fccw"
server_key: "2120afbcd5f66f89f347c1acc334a6ea"
server_key_iv: "05e97c202797d1408337ed8052c91ee3"
server_key_version: "encryptionkey@0"
I20260926 03:04:16.105294 18810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "63996c50a44f415f883a358f486327ee"
format_stamp: "Formatted at 2026-09-26 03:04:16 on dist-test-slave-fccw"
server_key: "2120afbcd5f66f89f347c1acc334a6ea"
server_key_iv: "05e97c202797d1408337ed8052c91ee3"
server_key_version: "encryptionkey@0"
I20260926 03:04:16.110498 18810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 03:04:16.113983 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:16.115247 18810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 03:04:16.115423 18810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "63996c50a44f415f883a358f486327ee"
format_stamp: "Formatted at 2026-09-26 03:04:16 on dist-test-slave-fccw"
server_key: "2120afbcd5f66f89f347c1acc334a6ea"
server_key_iv: "05e97c202797d1408337ed8052c91ee3"
server_key_version: "encryptionkey@0"
I20260926 03:04:16.115624 18810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:16.134958 18810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:16.138989 18810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:16.139274 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:16.140390 18810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:16.141881 18810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:16.141983 18810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:16.142129 18810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:16.142198 18810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:16.175807 18810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46783
I20260926 03:04:16.175905 18937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46783 every 8 connection(s)
I20260926 03:04:16.177212 18810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:04:16.186434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18810
I20260926 03:04:16.186643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:04:16.187175 388 external_mini_cluster.cc:1428] Setting key 0b0a8596ffdc45a3d96deb86e91e8cc0
Sep 26 03:04:16 dist-test-slave-fccw krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:16.200176 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.179653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39485 (local address 127.0.97.62:33277)
0926 03:04:16.179836 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.179840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.181310 (+ 1470us) server_negotiation.cc:407] Connection header received
0926 03:04:16.183149 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.183156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.183254 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.183367 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:16.185058 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.185749 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.188096 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.188577 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.188724 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.192761 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:16.192770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:16.192777 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:16.192825 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:16.196367 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:16.197338 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:16.197358 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:16.197364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:16.197586 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:16.198231 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:16.198235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:16.198237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:16.198502 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:16.198674 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.199427 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.199719 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 03:04:16.200881 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.180062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33277 (local address 127.0.97.2:39485)
0926 03:04:16.181137 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:16.181181 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:16.182871 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:16.183611 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:16.183620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:16.184293 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:16.184864 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.184882 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.185909 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.185915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.187930 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.187943 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.188826 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.188833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.189149 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.189835 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:16.189871 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:16.192522 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:16.196505 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:16.196512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:16.196525 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:16.197199 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:16.197807 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:16.197815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:16.197822 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:16.198100 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:16.198649 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:16.198668 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:16.199036 (+ 368us) client_negotiation.cc:769] Sending connection context
0926 03:04:16.199277 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 03:04:16.199674 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":605,"threads_started":1}
I20260926 03:04:16.202626 18938 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33277
I20260926 03:04:16.202957 18938 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:16.203723 18938 heartbeater.cc:511] Master 127.0.97.62:33277 requested a full tablet report, sending...
I20260926 03:04:16.205312 18615 ts_manager.cc:194] Registered new tserver with Master: 63996c50a44f415f883a358f486327ee (127.0.97.2:46783)
I20260926 03:04:16.206650 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39485
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:04:16.244306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33277
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:16.571084 18943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:16.571429 18943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:16.571602 18943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:16.578778 18943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:16.578944 18943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:16.579136 18943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:04:16.589176 18943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:16.590502 18943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:16.591892 18943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:16.605738 18948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:16.605772 18951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:16.605695 18949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:16.607210 18943 server_base.cc:1034] running on GCE node
I20260926 03:04:16.607905 18943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:16.608799 18943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:16.610029 18943 hybrid_clock.cc:648] HybridClock initialized: now 1790391856609990 us; error 49 us; skew 500 ppm
I20260926 03:04:16.610391 18943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:16 dist-test-slave-fccw krb5kdc[18570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:16.617621 18943 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:04:16.619203 18943 webserver.cc:460] Webserver started at http://127.0.97.3:35133/ using document root <none> and password file <none>
I20260926 03:04:16.619938 18943 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:16.620078 18943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:16.620445 18943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:16.623555 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "7e05dff0dd2849feae5544a87fadbfc8"
format_stamp: "Formatted at 2026-09-26 03:04:16 on dist-test-slave-fccw"
server_key: "793a4077ad9902d28216e805602cf007"
server_key_iv: "b11c18ef797262b82e46f2d5d64a6087"
server_key_version: "encryptionkey@0"
I20260926 03:04:16.624338 18943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7e05dff0dd2849feae5544a87fadbfc8"
format_stamp: "Formatted at 2026-09-26 03:04:16 on dist-test-slave-fccw"
server_key: "793a4077ad9902d28216e805602cf007"
server_key_iv: "b11c18ef797262b82e46f2d5d64a6087"
server_key_version: "encryptionkey@0"
I20260926 03:04:16.629509 18943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:04:16.632747 18958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:16.634043 18943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:04:16.634220 18943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "7e05dff0dd2849feae5544a87fadbfc8"
format_stamp: "Formatted at 2026-09-26 03:04:16 on dist-test-slave-fccw"
server_key: "793a4077ad9902d28216e805602cf007"
server_key_iv: "b11c18ef797262b82e46f2d5d64a6087"
server_key_version: "encryptionkey@0"
I20260926 03:04:16.634428 18943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:16.650475 18943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:16.654263 18943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:16.654546 18943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:16.655505 18943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:16.656785 18943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:16.656883 18943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:16.657086 18943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:16.657176 18943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:16.690440 18943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34557
I20260926 03:04:16.690490 19070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34557 every 8 connection(s)
I20260926 03:04:16.691756 18943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:04:16.694157 18804 heartbeater.cc:503] Master 127.0.97.62:33277 was elected leader, sending a full tablet report...
I20260926 03:04:16.697316 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 18943
I20260926 03:04:16.697494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:04:16.697956 388 external_mini_cluster.cc:1428] Setting key 53106a5d87b328f8a83cc22f4a06da2d
Sep 26 03:04:16 dist-test-slave-fccw krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:16.711226 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.694390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35813 (local address 127.0.97.62:33277)
0926 03:04:16.694606 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.694612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.695863 (+ 1251us) server_negotiation.cc:407] Connection header received
0926 03:04:16.697893 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.697899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.697986 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.698095 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:16.699592 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.700377 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.701992 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.702384 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.702512 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.705848 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:16.705853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:16.705856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:16.705886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:16.708418 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:16.709141 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:16.709145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:16.709147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:16.709290 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:16.709666 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:16.709668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:16.709670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:16.709920 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:16.710025 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.710690 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.710855 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:04:16.712390 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.694900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33277 (local address 127.0.97.3:35813)
0926 03:04:16.695700 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:16.695751 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:16.697547 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:16.698365 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:16.698372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:16.698905 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:16.699425 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.699442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.700530 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.700537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.701841 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.701852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.702613 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.702621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.702855 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.703455 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:16.703481 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:16.705644 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:16.708531 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:16.708537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:16.708548 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:16.708946 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:16.709379 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:16.709382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:16.709385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:16.709575 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:16.710017 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:16.710022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:16.710350 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 03:04:16.710567 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 03:04:16.711047 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":372,"threads_started":1}
I20260926 03:04:16.713637 19071 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33277
I20260926 03:04:16.713944 19071 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:16.714612 19071 heartbeater.cc:511] Master 127.0.97.62:33277 requested a full tablet report, sending...
I20260926 03:04:16.715876 18615 ts_manager.cc:194] Registered new tserver with Master: 7e05dff0dd2849feae5544a87fadbfc8 (127.0.97.3:34557)
I20260926 03:04:16.716652 18615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35813
I20260926 03:04:16.724887 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:04:16.741248 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.728796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54596 (local address 127.0.97.62:33277)
0926 03:04:16.729030 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.729040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.729363 (+ 323us) server_negotiation.cc:407] Connection header received
0926 03:04:16.729487 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.729491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.729543 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.729619 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:16.730981 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.731711 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.732941 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.733394 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.733485 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.735932 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:16.735936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:16.735940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:16.735968 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:16.738653 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:16.739403 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:16.739406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:16.739409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:16.739521 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:16.739916 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:16.739919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:16.739921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:16.740215 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:16.740426 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.740751 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.740881 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:04:16.745187 18615 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:04:16.752023 18615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:04:16.788800 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.764279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34557 (local address 127.0.0.1:42390)
0926 03:04:16.773856 (+ 9577us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:16.773904 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:16.774105 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:16.781660 (+ 7555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:16.781666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:16.781698 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:16.782162 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.782174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.784029 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.784035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.785932 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.785944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.787759 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.787766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.787937 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.788006 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 03:04:16.788112 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:04:16.788356 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9239,"mutex_wait_us":79,"thread_start_us":9067,"threads_started":1}
I20260926 03:04:16.789285 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.765332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.0.97.2:46783)
0926 03:04:16.780178 (+ 14846us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.780190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.780232 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:04:16.780360 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.780368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.780680 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.780900 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:16.781757 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.783244 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.786407 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.787479 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.787641 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.787863 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.788334 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.788913 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14673,"thread_start_us":14533,"threads_started":1}
I20260926 03:04:16.789305 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.764933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.0.97.3:34557)
0926 03:04:16.780957 (+ 16024us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.780967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.781047 (+ 80us) server_negotiation.cc:407] Connection header received
0926 03:04:16.781142 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.781149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.781395 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.781642 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:16.782356 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.783891 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.786080 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.787517 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.788280 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.788506 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.788573 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.788727 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15838,"thread_start_us":15661,"threads_started":1}
I20260926 03:04:16.789508 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.764707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46783 (local address 127.0.0.1:44664)
0926 03:04:16.779792 (+ 15085us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:16.779813 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:16.779960 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:16.780914 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:16.780921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:16.780950 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:16.781510 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.781526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.783408 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.783416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.786193 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.786212 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.787687 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.787694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.787914 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.787994 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 03:04:16.788117 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 03:04:16.789123 (+ 1006us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14945,"mutex_wait_us":822,"thread_start_us":14475,"threads_started":1}
I20260926 03:04:16.790381 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.764369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.0.97.1:45275)
0926 03:04:16.780651 (+ 16282us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.780671 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.780699 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:04:16.780785 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.780791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.781086 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.781295 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:16.782046 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.783664 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.786351 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.787750 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.787879 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.788103 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.789983 (+ 1880us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.790160 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16063,"thread_start_us":15861,"threads_started":1}
I20260926 03:04:16.790668 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.764306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55472)
0926 03:04:16.779512 (+ 15206us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:16.779535 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:16.779672 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:16.781301 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:16.781309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:16.781340 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:16.781842 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.781855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.783797 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.783804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.786215 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.786224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.788021 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.788027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.788187 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.788218 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:04:16.788297 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 03:04:16.788423 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15030,"mutex_wait_us":116,"thread_start_us":15208,"threads_started":1}
I20260926 03:04:16.793790 18739 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8076e431f9482a8ff04ed03fd045e5 (DEFAULT_TABLE table=test-table [id=c20bf526036646fb971720e2bb6dab1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:16.793812 18873 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8076e431f9482a8ff04ed03fd045e5 (DEFAULT_TABLE table=test-table [id=c20bf526036646fb971720e2bb6dab1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:16.793812 19006 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8076e431f9482a8ff04ed03fd045e5 (DEFAULT_TABLE table=test-table [id=c20bf526036646fb971720e2bb6dab1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:16.795116 18739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8076e431f9482a8ff04ed03fd045e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:16.795190 18873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8076e431f9482a8ff04ed03fd045e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:16.795190 19006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8076e431f9482a8ff04ed03fd045e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:16.806676 19089 tablet_bootstrap.cc:492] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Bootstrap starting.
I20260926 03:04:16.809759 19090 tablet_bootstrap.cc:492] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Bootstrap starting.
I20260926 03:04:16.810367 19089 tablet_bootstrap.cc:654] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:16.811342 19091 tablet_bootstrap.cc:492] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Bootstrap starting.
I20260926 03:04:16.812131 19089 log.cc:824] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:16.813552 19090 tablet_bootstrap.cc:654] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:16.814620 19091 tablet_bootstrap.cc:654] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:16.814766 19090 log.cc:824] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:16.815467 19089 tablet_bootstrap.cc:492] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: No bootstrap required, opened a new log
I20260926 03:04:16.815717 19091 log.cc:824] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:16.815742 19089 ts_tablet_manager.cc:1397] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:04:16.818209 19090 tablet_bootstrap.cc:492] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: No bootstrap required, opened a new log
I20260926 03:04:16.818480 19090 ts_tablet_manager.cc:1397] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:04:16.826534 19089 raft_consensus.cc:348] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.826997 19089 raft_consensus.cc:374] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:16.827101 19089 raft_consensus.cc:729] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74f28ff05b346cab02faeff58e5cb37, State: Initialized, Role: FOLLOWER
I20260926 03:04:16.827765 19089 consensus_queue.cc:260] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.828006 19091 tablet_bootstrap.cc:492] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: No bootstrap required, opened a new log
I20260926 03:04:16.828318 19091 ts_tablet_manager.cc:1397] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260926 03:04:16.828979 19090 raft_consensus.cc:348] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.829458 19090 raft_consensus.cc:374] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:16.829552 19090 raft_consensus.cc:729] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63996c50a44f415f883a358f486327ee, State: Initialized, Role: FOLLOWER
I20260926 03:04:16.829640 19089 ts_tablet_manager.cc:1428] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260926 03:04:16.830226 19090 consensus_queue.cc:260] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.832093 18938 heartbeater.cc:503] Master 127.0.97.62:33277 was elected leader, sending a full tablet report...
I20260926 03:04:16.832643 19090 ts_tablet_manager.cc:1428] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260926 03:04:16.838672 19091 raft_consensus.cc:348] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.839126 19091 raft_consensus.cc:374] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:16.839257 19091 raft_consensus.cc:729] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e05dff0dd2849feae5544a87fadbfc8, State: Initialized, Role: FOLLOWER
I20260926 03:04:16.839838 19091 consensus_queue.cc:260] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.840807 19071 heartbeater.cc:503] Master 127.0.97.62:33277 was elected leader, sending a full tablet report...
I20260926 03:04:16.841326 19091 ts_tablet_manager.cc:1428] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260926 03:04:16.914196 18805 tablet.cc:2369] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:04:16.929262 18939 tablet.cc:2369] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:04:16.943428 19072 tablet.cc:2369] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:04:16.978478 19097 raft_consensus.cc:482] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:04:16.978809 19097 raft_consensus.cc:504] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.980609 19097 leader_election.cc:288] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c74f28ff05b346cab02faeff58e5cb37 (127.0.97.1:45275), 63996c50a44f415f883a358f486327ee (127.0.97.2:46783)
I20260926 03:04:16.987064 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.981033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49613 (local address 127.0.97.1:45275)
0926 03:04:16.981233 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.981240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.981397 (+ 157us) server_negotiation.cc:407] Connection header received
0926 03:04:16.981600 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.981613 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.981669 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.981767 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:16.982215 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.983188 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.984967 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.986139 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.986243 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.986364 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.986683 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.986795 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:04:16.987092 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.980880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.97.3:49613)
0926 03:04:16.981275 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:16.981302 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:16.981461 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:16.981789 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:16.981796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:16.981822 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:16.982038 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.982049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.983325 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.983331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.984806 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.984818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.986322 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.986331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.986520 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.986559 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:04:16.986661 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:04:16.986803 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260926 03:04:16.987677 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.981133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46625 (local address 127.0.97.2:46783)
0926 03:04:16.981362 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:16.981370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:16.982055 (+ 685us) server_negotiation.cc:407] Connection header received
0926 03:04:16.982202 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:16.982206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:16.982266 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:16.982333 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:16.982871 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.983774 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.985694 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.986710 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.986826 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.986912 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:16.987322 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 03:04:16.987466 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 03:04:16.988273 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:16.981076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46783 (local address 127.0.97.3:46625)
0926 03:04:16.981942 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:16.981957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:16.982064 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:16.982375 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:16.982379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:16.982403 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:16.982726 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.982737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.983937 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.983940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.985538 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:16.985553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:16.986916 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:16.986926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:16.987108 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:16.987144 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:04:16.987822 (+ 678us) client_negotiation.cc:240] Negotiation successful
0926 03:04:16.987960 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":498,"threads_started":1}
I20260926 03:04:16.988281 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8076e431f9482a8ff04ed03fd045e5" candidate_uuid: "7e05dff0dd2849feae5544a87fadbfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74f28ff05b346cab02faeff58e5cb37" is_pre_election: true
I20260926 03:04:16.988921 18759 raft_consensus.cc:2454] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e05dff0dd2849feae5544a87fadbfc8 in term 0.
I20260926 03:04:16.989274 18893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8076e431f9482a8ff04ed03fd045e5" candidate_uuid: "7e05dff0dd2849feae5544a87fadbfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63996c50a44f415f883a358f486327ee" is_pre_election: true
I20260926 03:04:16.989681 18962 leader_election.cc:302] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e05dff0dd2849feae5544a87fadbfc8, c74f28ff05b346cab02faeff58e5cb37; no voters:
I20260926 03:04:16.989746 18893 raft_consensus.cc:2454] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e05dff0dd2849feae5544a87fadbfc8 in term 0.
I20260926 03:04:16.990098 19097 raft_consensus.cc:2781] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:04:16.990258 19097 raft_consensus.cc:482] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:04:16.990370 19097 raft_consensus.cc:3037] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:16.992724 19097 raft_consensus.cc:504] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:16.993395 19097 leader_election.cc:288] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [CANDIDATE]: Term 1 election: Requested vote from peers c74f28ff05b346cab02faeff58e5cb37 (127.0.97.1:45275), 63996c50a44f415f883a358f486327ee (127.0.97.2:46783)
I20260926 03:04:16.993782 18893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8076e431f9482a8ff04ed03fd045e5" candidate_uuid: "7e05dff0dd2849feae5544a87fadbfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63996c50a44f415f883a358f486327ee"
I20260926 03:04:16.993782 18759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8076e431f9482a8ff04ed03fd045e5" candidate_uuid: "7e05dff0dd2849feae5544a87fadbfc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74f28ff05b346cab02faeff58e5cb37"
I20260926 03:04:16.993995 18759 raft_consensus.cc:3037] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:16.993995 18893 raft_consensus.cc:3037] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:16.997371 18759 raft_consensus.cc:2454] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e05dff0dd2849feae5544a87fadbfc8 in term 1.
I20260926 03:04:16.997373 18893 raft_consensus.cc:2454] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e05dff0dd2849feae5544a87fadbfc8 in term 1.
I20260926 03:04:16.997776 18962 leader_election.cc:302] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e05dff0dd2849feae5544a87fadbfc8, c74f28ff05b346cab02faeff58e5cb37; no voters:
I20260926 03:04:16.998044 19097 raft_consensus.cc:2781] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:16.998453 19097 raft_consensus.cc:686] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 1 LEADER]: Becoming Leader. State: Replica: 7e05dff0dd2849feae5544a87fadbfc8, State: Running, Role: LEADER
I20260926 03:04:16.998929 19097 consensus_queue.cc:237] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } }
I20260926 03:04:17.003293 18615 catalog_manager.cc:5495] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e05dff0dd2849feae5544a87fadbfc8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7e05dff0dd2849feae5544a87fadbfc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e05dff0dd2849feae5544a87fadbfc8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 } health_report { overall_health: UNKNOWN } } }
I20260926 03:04:17.073661 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.068222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.0.97.3:34557)
0926 03:04:17.068435 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:17.068441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:17.068463 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:04:17.068576 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:17.068581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:17.068656 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:17.068816 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:04:17.069259 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.069961 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.071371 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.071767 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.071907 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:17.072520 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:04:17.073078 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:04:17.073267 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:17.073386 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:04:17.073434 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:04:17.081833 18893 raft_consensus.cc:1261] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 1 FOLLOWER]: Refusing update from remote peer 7e05dff0dd2849feae5544a87fadbfc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:17.081833 18759 raft_consensus.cc:1261] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 1 FOLLOWER]: Refusing update from remote peer 7e05dff0dd2849feae5544a87fadbfc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:17.082942 19097 consensus_queue.cc:1035] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63996c50a44f415f883a358f486327ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:17.083232 19099 consensus_queue.cc:1035] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74f28ff05b346cab02faeff58e5cb37" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:17.102816 19102 mvcc.cc:205] Tried to move back new op lower bound from 7333445046596485120 to 7333445046271619072. Current Snapshot: MvccSnapshot[applied={T|T < 7333445046596485120}]
I20260926 03:04:17.116565 18614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:17.116894 18614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:17.120806 18614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 240a413ed0ab481884b8147d2991ead6: Sending DeleteTablet for 3 replicas of tablet 3b8076e431f9482a8ff04ed03fd045e5
I20260926 03:04:17.122023 18739 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8076e431f9482a8ff04ed03fd045e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55472
I20260926 03:04:17.122080 18873 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8076e431f9482a8ff04ed03fd045e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44664
I20260926 03:04:17.122852 19006 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b8076e431f9482a8ff04ed03fd045e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42390
I20260926 03:04:17.123046 19112 tablet_replica.cc:330] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: stopping tablet replica
I20260926 03:04:17.123441 19112 raft_consensus.cc:2229] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:17.124220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18676
I20260926 03:04:17.125491 19112 raft_consensus.cc:2258] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:17.133929 19112 ts_tablet_manager.cc:1905] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:04:17 dist-test-slave-fccw krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:04:17.144399 19114 tablet_replica.cc:330] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: stopping tablet replica
I20260926 03:04:17.148452 19112 ts_tablet_manager.cc:1918] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:17.149127 19114 raft_consensus.cc:2229] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:04:17.149676 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.127103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.0.97.1:45275)
0926 03:04:17.127318 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:17.127333 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:17.127787 (+ 454us) server_negotiation.cc:407] Connection header received
0926 03:04:17.128130 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:17.128137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:17.128230 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:17.128311 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:17.131005 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.132323 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.134222 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.135095 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.135217 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:17.141538 (+ 6321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:17.141553 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:17.141571 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:17.141616 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:17.145873 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:17.146815 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:17.146826 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:17.146829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:17.146995 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:17.147481 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:17.147487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:17.147490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:17.147925 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:17.148226 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:17.148874 (+ 648us) server_negotiation.cc:299] Negotiation successful
0926 03:04:17.149177 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:04:17.150089 19114 raft_consensus.cc:2258] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:17.151079 18799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:17.152899 19114 ts_tablet_manager.cc:1905] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:17.153681 19113 tablet_replica.cc:330] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: stopping tablet replica
I20260926 03:04:17.154224 19113 raft_consensus.cc:2229] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:17.154819 19113 raft_consensus.cc:2258] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:17.157244 19112 log.cc:1196] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/3b8076e431f9482a8ff04ed03fd045e5
I20260926 03:04:17.159104 19113 ts_tablet_manager.cc:1905] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:17.165179 19114 ts_tablet_manager.cc:1918] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:17.165426 19114 log.cc:1196] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/3b8076e431f9482a8ff04ed03fd045e5
I20260926 03:04:17.166100 19114 ts_tablet_manager.cc:1939] T 3b8076e431f9482a8ff04ed03fd045e5 P 7e05dff0dd2849feae5544a87fadbfc8: Deleting consensus metadata
I20260926 03:04:17.166451 19113 ts_tablet_manager.cc:1918] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:17.166653 19113 log.cc:1196] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/3b8076e431f9482a8ff04ed03fd045e5
I20260926 03:04:17.167246 19113 ts_tablet_manager.cc:1939] T 3b8076e431f9482a8ff04ed03fd045e5 P 63996c50a44f415f883a358f486327ee: Deleting consensus metadata
I20260926 03:04:17.168609 18599 catalog_manager.cc:4841] TS 7e05dff0dd2849feae5544a87fadbfc8 (127.0.97.3:34557): tablet 3b8076e431f9482a8ff04ed03fd045e5 (table test-table [id=c20bf526036646fb971720e2bb6dab1f]) successfully deleted
I20260926 03:04:17.169145 18600 catalog_manager.cc:4841] TS 63996c50a44f415f883a358f486327ee (127.0.97.2:46783): tablet 3b8076e431f9482a8ff04ed03fd045e5 (table test-table [id=c20bf526036646fb971720e2bb6dab1f]) successfully deleted
I20260926 03:04:17.273921 19112 ts_tablet_manager.cc:1939] T 3b8076e431f9482a8ff04ed03fd045e5 P c74f28ff05b346cab02faeff58e5cb37: Deleting consensus metadata
I20260926 03:04:17.273973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18676
W20260926 03:04:17.292052 18602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45275 (error 108)
I20260926 03:04:17.292790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18810
I20260926 03:04:17.294631 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.293481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55488)
0926 03:04:17.294312 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:17.294431 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":599,"threads_started":1}
W20260926 03:04:17.294992 18602 catalog_manager.cc:4584] TS c74f28ff05b346cab02faeff58e5cb37 (127.0.97.1:45275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b8076e431f9482a8ff04ed03fd045e5: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Sep 26 03:04:17 dist-test-slave-fccw krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:04:17.308737 19088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.293692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.0.97.2:46783)
0926 03:04:17.293938 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:17.293946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:17.293973 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:04:17.294060 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:17.294069 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:17.294153 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:17.294259 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:17.295933 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.296727 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.298127 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.298527 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.298664 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:17.302852 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:17.302867 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:17.302886 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:17.302925 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:17.305932 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:17.306650 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:17.306658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:17.306660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:17.306778 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:17.307162 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:17.307167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:17.307171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:17.307564 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:17.307858 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:17.308185 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 03:04:17.308407 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:04:17.309537 18933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:17.348881 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.348346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55504)
0926 03:04:17.348593 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:17.348699 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 03:04:17.350095 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.349644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55516)
0926 03:04:17.349831 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:17.349928 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 03:04:17.399574 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18810
I20260926 03:04:17.410828 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.410223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55526)
0926 03:04:17.410496 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:17.410613 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 03:04:17.412039 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.411561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55538)
0926 03:04:17.411756 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:17.411863 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 03:04:17.417217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18943
Sep 26 03:04:17 dist-test-slave-fccw krb5kdc[18570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:04:17.434778 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.418010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.0.97.3:34557)
0926 03:04:17.418220 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:17.418227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:17.418249 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:04:17.418320 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:17.418325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:17.418409 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:17.418531 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:17.419762 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.421072 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.422664 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:17.423364 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:17.423506 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:17.428431 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:17.428445 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:17.428461 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:17.428499 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:17.431567 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:17.432353 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:17.432360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:17.432362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:17.432534 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:17.432951 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:17.432954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:17.432957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:17.433544 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:17.433805 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:17.434217 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 03:04:17.434401 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:04:17.435551 19066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:17.492916 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.492351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55542)
0926 03:04:17.492606 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:17.492718 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 03:04:17.494140 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:17.493653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45275 (local address 127.0.0.1:55556)
0926 03:04:17.493880 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:17.493975 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 03:04:17.559368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18943
I20260926 03:04:17.579365 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 18583
I20260926 03:04:17.580446 18645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:17.694100 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 18583
2026-09-26T03:04:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4401 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T03:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:17Z Disabled control of system clock
I20260926 03:04:17.769737 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36377
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:04:18.107810 19125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:18.108110 19125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:18.108256 19125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:18.115654 19125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:18.115761 19125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:18.115849 19125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:18.115957 19125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:18.116017 19125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:18.125270 19125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:18.126565 19125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:18.128430 19125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:18.140372 19130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:18.140430 19131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:18.140348 19133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:18.141410 19125 server_base.cc:1034] running on GCE node
I20260926 03:04:18.142208 19125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:18.143642 19125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:18.144886 19125 hybrid_clock.cc:648] HybridClock initialized: now 1790391858144851 us; error 65 us; skew 500 ppm
I20260926 03:04:18.145262 19125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:18.147466 19125 webserver.cc:460] Webserver started at http://127.0.97.62:38861/ using document root <none> and password file <none>
I20260926 03:04:18.148149 19125 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:18.148278 19125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:18.148603 19125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:18.151564 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "5013ffda0b5a43c195bfa8e71541a3b9"
format_stamp: "Formatted at 2026-09-26 03:04:18 on dist-test-slave-fccw"
server_key: "eb967208fb23430909d83dffd33afeed"
server_key_iv: "2cefa281e7a9307adec002c82df4493f"
server_key_version: "encryptionkey@0"
I20260926 03:04:18.152350 19125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "5013ffda0b5a43c195bfa8e71541a3b9"
format_stamp: "Formatted at 2026-09-26 03:04:18 on dist-test-slave-fccw"
server_key: "eb967208fb23430909d83dffd33afeed"
server_key_iv: "2cefa281e7a9307adec002c82df4493f"
server_key_version: "encryptionkey@0"
I20260926 03:04:18.157330 19125 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:04:18.161399 19139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:18.162899 19125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:18.163085 19125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "5013ffda0b5a43c195bfa8e71541a3b9"
format_stamp: "Formatted at 2026-09-26 03:04:18 on dist-test-slave-fccw"
server_key: "eb967208fb23430909d83dffd33afeed"
server_key_iv: "2cefa281e7a9307adec002c82df4493f"
server_key_version: "encryptionkey@0"
I20260926 03:04:18.163307 19125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:18.178464 19125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:18.179314 19125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:18.179616 19125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:18.198392 19125 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36377
I20260926 03:04:18.198421 19190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36377 every 8 connection(s)
I20260926 03:04:18.199791 19125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:18.202565 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 19125
I20260926 03:04:18.202751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:18.203217 388 external_mini_cluster.cc:1428] Setting key c1bc5822d109692323f217d5f910d4c7
I20260926 03:04:18.203766 19191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:18.216434 19191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9: Bootstrap starting.
I20260926 03:04:18.217351 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:18.205131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.0.97.62:36377)
0926 03:04:18.206077 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:18.206088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:18.206126 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:04:18.208443 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:18.208465 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:18.208938 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:18.209472 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:18.210043 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:18.211316 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:18.212659 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:18.213518 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:18.213737 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:18.214935 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:18.214943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:18.214956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:18.215489 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:18.215564 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:18.215584 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:18.215709 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:18.215936 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 03:04:18.216234 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":561,"thread_start_us":364,"threads_started":1}
I20260926 03:04:18.219650 19191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:18.220757 19191 log.cc:824] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:18.223474 19191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9: No bootstrap required, opened a new log
I20260926 03:04:18.231156 19191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260926 03:04:18.231520 19191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:18.231592 19191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5013ffda0b5a43c195bfa8e71541a3b9, State: Initialized, Role: FOLLOWER
I20260926 03:04:18.232091 19191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260926 03:04:18.232280 19191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:18.232374 19191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:18.232491 19191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:18.234644 19191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260926 03:04:18.235237 19191 leader_election.cc:302] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5013ffda0b5a43c195bfa8e71541a3b9; no voters:
I20260926 03:04:18.235909 19191 leader_election.cc:288] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:18.236032 19196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:18.236758 19196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5013ffda0b5a43c195bfa8e71541a3b9, State: Running, Role: LEADER
I20260926 03:04:18.237325 19196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } }
I20260926 03:04:18.238236 19191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:18.240432 19197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } }
I20260926 03:04:18.240800 19197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:18.243206 19198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5013ffda0b5a43c195bfa8e71541a3b9. Latest consensus state: current_term: 1 leader_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5013ffda0b5a43c195bfa8e71541a3b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36377 } } }
I20260926 03:04:18.243574 19198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:18.243804 19203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:18.248747 19203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:18.257433 19203 catalog_manager.cc:1347] Generated new cluster ID: dad352e2174949808e2827b230aeb7cd
I20260926 03:04:18.257550 19203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:18.290355 19203 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:18.292308 19203 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:18.308795 19203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9: Generated new TSK 0
I20260926 03:04:18.309725 19203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:18.322162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--builtin_ntp_servers=127.0.97.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:04:18.659086 19215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:18.659370 19215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:18.659513 19215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:18.667164 19215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:18.667309 19215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:18.667445 19215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:04:18.676780 19215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:18.678153 19215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:18.680028 19215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:18.693420 19221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:18.693361 19223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:18.693425 19220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:18.694101 19215 server_base.cc:1034] running on GCE node
I20260926 03:04:18.694723 19215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:18.695681 19215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:18.696919 19215 hybrid_clock.cc:648] HybridClock initialized: now 1790391858696869 us; error 57 us; skew 500 ppm
I20260926 03:04:18.697263 19215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:18.699369 19215 webserver.cc:460] Webserver started at http://127.0.97.1:33525/ using document root <none> and password file <none>
I20260926 03:04:18.700040 19215 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:18.700167 19215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:18.700510 19215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:18.703359 19215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "c87a6306c5654a878bf61c47030e1a48"
format_stamp: "Formatted at 2026-09-26 03:04:18 on dist-test-slave-fccw"
server_key: "42ee12b853df8e284e26756841e80b05"
server_key_iv: "652757a5bdff5ccce8fb99f6255719f5"
server_key_version: "encryptionkey@0"
I20260926 03:04:18.704173 19215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c87a6306c5654a878bf61c47030e1a48"
format_stamp: "Formatted at 2026-09-26 03:04:18 on dist-test-slave-fccw"
server_key: "42ee12b853df8e284e26756841e80b05"
server_key_iv: "652757a5bdff5ccce8fb99f6255719f5"
server_key_version: "encryptionkey@0"
I20260926 03:04:18.709187 19215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:04:18.712492 19229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:18.713800 19215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:04:18.713973 19215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "c87a6306c5654a878bf61c47030e1a48"
format_stamp: "Formatted at 2026-09-26 03:04:18 on dist-test-slave-fccw"
server_key: "42ee12b853df8e284e26756841e80b05"
server_key_iv: "652757a5bdff5ccce8fb99f6255719f5"
server_key_version: "encryptionkey@0"
I20260926 03:04:18.714186 19215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:18.741570 19215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:18.742443 19215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:18.742713 19215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:18.743762 19215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:18.745323 19215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:18.745478 19215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:18.745636 19215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:18.745726 19215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:18.778350 19215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42325
I20260926 03:04:18.778447 19341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42325 every 8 connection(s)
I20260926 03:04:18.779721 19215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:04:18.789376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 19215
I20260926 03:04:18.789552 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:04:18.790153 388 external_mini_cluster.cc:1428] Setting key 68c4389279f5a402640c5f426bc2212f
I20260926 03:04:18.792308 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:18.782219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55577 (local address 127.0.97.62:36377)
0926 03:04:18.782667 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:18.782671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:18.783708 (+ 1037us) server_negotiation.cc:407] Connection header received
0926 03:04:18.785574 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:18.785579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:18.785671 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:18.785789 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:18.786831 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:18.787563 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:18.789152 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:18.789689 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:18.789848 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:18.791454 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:18.791460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:18.791465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:18.791488 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:18.791535 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:18.791550 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:18.791643 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:18.791842 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 03:04:18.791944 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":247,"threads_started":1}
I20260926 03:04:18.793404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--builtin_ntp_servers=127.0.97.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:04:18.793587 19345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:18.782669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.97.1:55577)
0926 03:04:18.783550 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:18.783590 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:18.785306 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:18.786026 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:18.786034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:18.786118 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:18.786660 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:18.786675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:18.787726 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:18.787730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:18.788940 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:18.788951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:18.789952 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:18.789959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:18.790257 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:18.791079 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:18.791083 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:18.791145 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:18.791242 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:18.791649 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:18.791657 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:18.791663 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:04:18.791807 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 03:04:18.792184 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":292,"threads_started":1}
I20260926 03:04:18.795034 19342 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36377
I20260926 03:04:18.795318 19342 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:18.796038 19342 heartbeater.cc:511] Master 127.0.97.62:36377 requested a full tablet report, sending...
I20260926 03:04:18.798092 19156 ts_manager.cc:194] Registered new tserver with Master: c87a6306c5654a878bf61c47030e1a48 (127.0.97.1:42325)
I20260926 03:04:18.799552 19156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:55577
W20260926 03:04:19.130532 19346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:19.130806 19346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:19.130892 19346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:19.138160 19346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:19.138351 19346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:19.138553 19346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:04:19.147934 19346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:19.149313 19346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:19.151079 19346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:19.164137 19352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:19.164233 19354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:19.164179 19351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:19.165414 19346 server_base.cc:1034] running on GCE node
I20260926 03:04:19.166087 19346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:19.167101 19346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:19.168339 19346 hybrid_clock.cc:648] HybridClock initialized: now 1790391859168269 us; error 65 us; skew 500 ppm
I20260926 03:04:19.168679 19346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:19.170953 19346 webserver.cc:460] Webserver started at http://127.0.97.2:37545/ using document root <none> and password file <none>
I20260926 03:04:19.171649 19346 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:19.171778 19346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:19.172099 19346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:19.175235 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "774f388b003742b98c4281f83ca5ddde"
format_stamp: "Formatted at 2026-09-26 03:04:19 on dist-test-slave-fccw"
server_key: "047bc9eb8eb96bc3068a32a5da882bef"
server_key_iv: "2bcee2effb5b2ad99875b802ec326ad2"
server_key_version: "encryptionkey@0"
I20260926 03:04:19.176051 19346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "774f388b003742b98c4281f83ca5ddde"
format_stamp: "Formatted at 2026-09-26 03:04:19 on dist-test-slave-fccw"
server_key: "047bc9eb8eb96bc3068a32a5da882bef"
server_key_iv: "2bcee2effb5b2ad99875b802ec326ad2"
server_key_version: "encryptionkey@0"
I20260926 03:04:19.181279 19346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:04:19.184657 19360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:19.186028 19346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:04:19.186203 19346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "774f388b003742b98c4281f83ca5ddde"
format_stamp: "Formatted at 2026-09-26 03:04:19 on dist-test-slave-fccw"
server_key: "047bc9eb8eb96bc3068a32a5da882bef"
server_key_iv: "2bcee2effb5b2ad99875b802ec326ad2"
server_key_version: "encryptionkey@0"
I20260926 03:04:19.186443 19346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:19.206930 19346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:19.207754 19346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:19.208034 19346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:19.209092 19346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:19.210572 19346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:19.210697 19346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:19.210819 19346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:19.210906 19346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:19.243978 19346 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42303
I20260926 03:04:19.244064 19472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42303 every 8 connection(s)
I20260926 03:04:19.245330 19346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:04:19.249375 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 19346
I20260926 03:04:19.249558 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:04:19.250092 388 external_mini_cluster.cc:1428] Setting key 2e51e3c1a49341e92ca0188ff0a201c5
I20260926 03:04:19.253276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--builtin_ntp_servers=127.0.97.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:04:19.259661 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.248029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49343 (local address 127.0.97.62:36377)
0926 03:04:19.248220 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:19.248224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:19.249872 (+ 1648us) server_negotiation.cc:407] Connection header received
0926 03:04:19.252091 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:19.252096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:19.252177 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:19.252275 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:19.253718 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.254414 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.256816 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.257313 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.257409 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.258978 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:19.258982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:19.258985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:19.259003 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:19.259035 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:19.259045 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:19.259114 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:19.259362 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 03:04:19.259437 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:04:19.261150 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.248574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.97.2:49343)
0926 03:04:19.249691 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:19.249746 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:19.251811 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:19.252538 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:19.252550 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:19.252666 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:19.253556 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.253584 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.254569 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.254578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.256676 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.256692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.257528 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.257536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.257857 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.258606 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:19.258612 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:19.258667 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:19.258777 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:19.259142 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:19.259154 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:19.259165 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:04:19.259346 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 03:04:19.259763 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":301,"threads_started":1}
I20260926 03:04:19.262633 19473 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36377
I20260926 03:04:19.262955 19473 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:19.263684 19473 heartbeater.cc:511] Master 127.0.97.62:36377 requested a full tablet report, sending...
I20260926 03:04:19.265112 19156 ts_manager.cc:194] Registered new tserver with Master: 774f388b003742b98c4281f83ca5ddde (127.0.97.2:42303)
I20260926 03:04:19.265897 19156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49343
W20260926 03:04:19.591842 19476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:19.592195 19476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:19.592393 19476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:19.599686 19476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:19.599852 19476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:19.600034 19476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:04:19.609200 19476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:19.610520 19476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:19.612301 19476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:19.625547 19481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:19.625473 19482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:19.625509 19484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:19.626398 19476 server_base.cc:1034] running on GCE node
I20260926 03:04:19.626935 19476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:19.627826 19476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:19.629086 19476 hybrid_clock.cc:648] HybridClock initialized: now 1790391859629037 us; error 40 us; skew 500 ppm
I20260926 03:04:19.629410 19476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:19.631619 19476 webserver.cc:460] Webserver started at http://127.0.97.3:44475/ using document root <none> and password file <none>
I20260926 03:04:19.632320 19476 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:19.632436 19476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:19.632761 19476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:19.635645 19476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "da8685a991b54bd98fd75def08f047d8"
format_stamp: "Formatted at 2026-09-26 03:04:19 on dist-test-slave-fccw"
server_key: "e71370300da0530ea7df3308792a651d"
server_key_iv: "69a1bfe3dd3f6422a68d7a8758f3c0a8"
server_key_version: "encryptionkey@0"
I20260926 03:04:19.636418 19476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "da8685a991b54bd98fd75def08f047d8"
format_stamp: "Formatted at 2026-09-26 03:04:19 on dist-test-slave-fccw"
server_key: "e71370300da0530ea7df3308792a651d"
server_key_iv: "69a1bfe3dd3f6422a68d7a8758f3c0a8"
server_key_version: "encryptionkey@0"
I20260926 03:04:19.641546 19476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 03:04:19.644747 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:19.646205 19476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:04:19.646384 19476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "da8685a991b54bd98fd75def08f047d8"
format_stamp: "Formatted at 2026-09-26 03:04:19 on dist-test-slave-fccw"
server_key: "e71370300da0530ea7df3308792a651d"
server_key_iv: "69a1bfe3dd3f6422a68d7a8758f3c0a8"
server_key_version: "encryptionkey@0"
I20260926 03:04:19.646610 19476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:19.665834 19476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:19.666730 19476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:19.667016 19476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:19.668083 19476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:19.669572 19476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:19.669661 19476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:19.669754 19476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:19.669806 19476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:19.703234 19476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36167
I20260926 03:04:19.703284 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36167 every 8 connection(s)
I20260926 03:04:19.704573 19476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:04:19.711190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 19476
I20260926 03:04:19.711386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:04:19.711812 388 external_mini_cluster.cc:1428] Setting key cd395a1a278a79248df5192253004f37
I20260926 03:04:19.717284 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.707245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47431 (local address 127.0.97.62:36377)
0926 03:04:19.707483 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:19.707489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:19.708876 (+ 1387us) server_negotiation.cc:407] Connection header received
0926 03:04:19.710769 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:19.710775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:19.710882 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:19.711008 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:19.712155 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.712945 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.714615 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.715034 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.715171 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.716461 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:19.716466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:19.716469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:19.716485 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:19.716515 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:19.716526 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:19.716619 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:19.716842 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 03:04:19.716936 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:04:19.718438 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.707839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36377 (local address 127.0.97.3:47431)
0926 03:04:19.708723 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:19.708758 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:19.710482 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:19.711191 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:19.711199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:19.711286 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:19.711986 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.712005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.713144 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.713149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.714467 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.714480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.715259 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.715266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.715509 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.716132 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:19.716135 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:19.716180 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:19.716261 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:19.716616 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:19.716628 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:19.716639 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 03:04:19.716834 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 03:04:19.717296 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":276,"threads_started":1}
I20260926 03:04:19.719738 19603 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36377
I20260926 03:04:19.720027 19603 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:19.720782 19603 heartbeater.cc:511] Master 127.0.97.62:36377 requested a full tablet report, sending...
I20260926 03:04:19.722147 19156 ts_manager.cc:194] Registered new tserver with Master: da8685a991b54bd98fd75def08f047d8 (127.0.97.3:36167)
I20260926 03:04:19.722905 19156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:47431
I20260926 03:04:19.726753 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:04:19.736527 19344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.730872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.0.97.62:36377)
0926 03:04:19.731027 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:19.731031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:19.731476 (+ 445us) server_negotiation.cc:407] Connection header received
0926 03:04:19.731610 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:19.731614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:19.731680 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:19.731761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:19.732221 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.732919 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.734292 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.734681 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.734775 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.735921 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:19.735925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:19.735929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:19.735946 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:19.735974 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:19.735985 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:19.736096 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:19.736254 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 03:04:19.736297 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:04:19.740593 19156 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:04:19.747637 19156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:04:19.771214 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.760182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.0.97.3:36167)
0926 03:04:19.761554 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:19.761584 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:19.762260 (+ 676us) server_negotiation.cc:407] Connection header received
0926 03:04:19.762546 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:19.762552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:19.762756 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:19.762942 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:19.763821 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.765253 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.768416 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.769907 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.770033 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.770322 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:19.770664 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 03:04:19.770861 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":1069,"threads_started":1}
I20260926 03:04:19.771409 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.760081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36167 (local address 127.0.0.1:37838)
0926 03:04:19.762117 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:19.762155 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:19.762344 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:19.762955 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:19.762962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:19.762994 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:19.763595 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.763609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.765385 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.765393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.768133 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.768146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.770109 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.770115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.770375 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.770452 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 03:04:19.770607 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 03:04:19.770964 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1851,"mutex_wait_us":172,"thread_start_us":1669,"threads_started":1}
I20260926 03:04:19.774842 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.760972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.0.97.2:42303)
0926 03:04:19.766846 (+ 5874us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:19.766855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:19.766886 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:04:19.766999 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:19.767005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:19.767274 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:19.767555 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:19.768634 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.770129 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.772699 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.773827 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.773972 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.774219 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:19.774371 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:04:19.774527 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5687,"thread_start_us":5566,"threads_started":1}
I20260926 03:04:19.774860 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.760329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42303 (local address 127.0.0.1:55064)
0926 03:04:19.765871 (+ 5542us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:19.765895 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:19.766009 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:19.767764 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:19.767768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:19.767791 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:19.768448 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.768464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.770298 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.770304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.772472 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.772486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.774023 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.774029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.774189 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.774219 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:04:19.774317 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 03:04:19.774446 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5373,"mutex_wait_us":7,"thread_start_us":6404,"threads_started":1}
I20260926 03:04:19.775488 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.763971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42325 (local address 127.0.0.1:60620)
0926 03:04:19.766870 (+ 2899us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:19.766887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:19.767001 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:19.767879 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:19.767884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:19.767918 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:19.768498 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.768507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.769854 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.769862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.772595 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:19.772685 (+ 90us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.774758 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.774764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:19.774944 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.774974 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:04:19.775061 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:04:19.775174 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2752,"mutex_wait_us":321,"thread_start_us":1567,"threads_started":1}
I20260926 03:04:19.776206 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:19.763621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.0.97.1:42325)
0926 03:04:19.766464 (+ 2843us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:19.766473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:19.767129 (+ 656us) server_negotiation.cc:407] Connection header received
0926 03:04:19.767244 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:19.767251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:19.767458 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:19.767642 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:19.768672 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.769714 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.772863 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:19.774282 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:19.774420 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:19.774620 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:19.775529 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 03:04:19.775863 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2693,"thread_start_us":2572,"threads_started":1}
I20260926 03:04:19.775900 19538 tablet_service.cc:1467] Processing CreateTablet for tablet 2e56d7f951354fac9b458ad6f57f13a2 (DEFAULT_TABLE table=test-table [id=14d35b227f1d4367b1f758ffe9e86e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:19.777642 19538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e56d7f951354fac9b458ad6f57f13a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:19.778527 19408 tablet_service.cc:1467] Processing CreateTablet for tablet 2e56d7f951354fac9b458ad6f57f13a2 (DEFAULT_TABLE table=test-table [id=14d35b227f1d4367b1f758ffe9e86e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:19.779577 19408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e56d7f951354fac9b458ad6f57f13a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:19.779714 19277 tablet_service.cc:1467] Processing CreateTablet for tablet 2e56d7f951354fac9b458ad6f57f13a2 (DEFAULT_TABLE table=test-table [id=14d35b227f1d4367b1f758ffe9e86e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:19.780968 19277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e56d7f951354fac9b458ad6f57f13a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:19.796030 19622 tablet_bootstrap.cc:492] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Bootstrap starting.
I20260926 03:04:19.799780 19622 tablet_bootstrap.cc:654] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:19.800814 19622 log.cc:824] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:19.801843 19621 tablet_bootstrap.cc:492] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Bootstrap starting.
I20260926 03:04:19.802526 19623 tablet_bootstrap.cc:492] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Bootstrap starting.
I20260926 03:04:19.803200 19342 heartbeater.cc:503] Master 127.0.97.62:36377 was elected leader, sending a full tablet report...
I20260926 03:04:19.805459 19621 tablet_bootstrap.cc:654] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:19.805963 19622 tablet_bootstrap.cc:492] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: No bootstrap required, opened a new log
I20260926 03:04:19.806229 19622 ts_tablet_manager.cc:1397] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260926 03:04:19.806486 19623 tablet_bootstrap.cc:654] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:19.806674 19621 log.cc:824] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:19.807663 19623 log.cc:824] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:19.812306 19623 tablet_bootstrap.cc:492] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: No bootstrap required, opened a new log
I20260926 03:04:19.812588 19623 ts_tablet_manager.cc:1397] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 03:04:19.812594 19621 tablet_bootstrap.cc:492] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: No bootstrap required, opened a new log
I20260926 03:04:19.812853 19621 ts_tablet_manager.cc:1397] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 03:04:19.817361 19622 raft_consensus.cc:348] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:19.817812 19622 raft_consensus.cc:374] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:19.817931 19622 raft_consensus.cc:729] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774f388b003742b98c4281f83ca5ddde, State: Initialized, Role: FOLLOWER
I20260926 03:04:19.818598 19622 consensus_queue.cc:260] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:19.819862 19473 heartbeater.cc:503] Master 127.0.97.62:36377 was elected leader, sending a full tablet report...
I20260926 03:04:19.820596 19622 ts_tablet_manager.cc:1428] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260926 03:04:19.822182 19623 raft_consensus.cc:348] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:19.822604 19623 raft_consensus.cc:374] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:19.822701 19623 raft_consensus.cc:729] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87a6306c5654a878bf61c47030e1a48, State: Initialized, Role: FOLLOWER
I20260926 03:04:19.823297 19623 consensus_queue.cc:260] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:19.823252 19621 raft_consensus.cc:348] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:19.823661 19621 raft_consensus.cc:374] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:19.823776 19621 raft_consensus.cc:729] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8685a991b54bd98fd75def08f047d8, State: Initialized, Role: FOLLOWER
I20260926 03:04:19.824481 19621 consensus_queue.cc:260] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:19.825191 19623 ts_tablet_manager.cc:1428] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260926 03:04:19.825627 19603 heartbeater.cc:503] Master 127.0.97.62:36377 was elected leader, sending a full tablet report...
I20260926 03:04:19.826145 19621 ts_tablet_manager.cc:1428] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
W20260926 03:04:19.956482 19604 tablet.cc:2369] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:04:19.997462 19474 tablet.cc:2369] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:04:19.998440 19627 raft_consensus.cc:482] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:04:19.998684 19627 raft_consensus.cc:504] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:20.000211 19627 leader_election.cc:288] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da8685a991b54bd98fd75def08f047d8 (127.0.97.3:36167), c87a6306c5654a878bf61c47030e1a48 (127.0.97.1:42325)
I20260926 03:04:20.008854 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.000650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49017 (local address 127.0.97.3:36167)
0926 03:04:20.000821 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:20.000835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:20.001453 (+ 618us) server_negotiation.cc:407] Connection header received
0926 03:04:20.001718 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:20.001724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:20.001802 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:20.001870 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:20.002748 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.003870 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.006859 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.007882 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.007994 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.008066 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:20.008466 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 03:04:20.008605 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:04:20.008926 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.000649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36167 (local address 127.0.97.2:49017)
0926 03:04:20.001343 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:20.001359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:20.001531 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:20.002105 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:20.002111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:20.002181 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:20.002595 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:20.002610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.003994 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.003997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:20.006627 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:20.006659 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.008084 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.008090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:20.008291 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.008328 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 03:04:20.008428 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:04:20.008522 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":157,"spinlock_wait_cycles":57600,"thread_start_us":402,"threads_started":1}
I20260926 03:04:20.009069 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.001445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42325 (local address 127.0.97.2:56129)
0926 03:04:20.002018 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:20.002032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:20.002150 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:20.002461 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:20.002463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:20.002484 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:20.002827 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:20.002834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.004110 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.004117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:20.006633 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:20.006659 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.008279 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.008283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:20.008479 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.008511 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:04:20.008597 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 03:04:20.008708 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":171,"thread_start_us":295,"threads_started":1}
I20260926 03:04:20.009657 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.001728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56129 (local address 127.0.97.1:42325)
0926 03:04:20.001929 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:20.001934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:20.002124 (+ 190us) server_negotiation.cc:407] Connection header received
0926 03:04:20.002249 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:20.002256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:20.002347 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:20.002444 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:20.002976 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.003977 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.006896 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.008104 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.008234 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.008332 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:20.008649 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 03:04:20.008772 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:04:20.010036 19558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e56d7f951354fac9b458ad6f57f13a2" candidate_uuid: "774f388b003742b98c4281f83ca5ddde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8685a991b54bd98fd75def08f047d8" is_pre_election: true
I20260926 03:04:20.010533 19558 raft_consensus.cc:2454] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774f388b003742b98c4281f83ca5ddde in term 0.
I20260926 03:04:20.010548 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e56d7f951354fac9b458ad6f57f13a2" candidate_uuid: "774f388b003742b98c4281f83ca5ddde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87a6306c5654a878bf61c47030e1a48" is_pre_election: true
I20260926 03:04:20.011051 19297 raft_consensus.cc:2454] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774f388b003742b98c4281f83ca5ddde in term 0.
I20260926 03:04:20.011646 19364 leader_election.cc:302] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774f388b003742b98c4281f83ca5ddde, da8685a991b54bd98fd75def08f047d8; no voters:
I20260926 03:04:20.012020 19627 raft_consensus.cc:2781] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:04:20.012137 19627 raft_consensus.cc:482] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:04:20.012279 19627 raft_consensus.cc:3037] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:20.014532 19627 raft_consensus.cc:504] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:20.015087 19627 leader_election.cc:288] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [CANDIDATE]: Term 1 election: Requested vote from peers da8685a991b54bd98fd75def08f047d8 (127.0.97.3:36167), c87a6306c5654a878bf61c47030e1a48 (127.0.97.1:42325)
I20260926 03:04:20.015549 19558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e56d7f951354fac9b458ad6f57f13a2" candidate_uuid: "774f388b003742b98c4281f83ca5ddde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da8685a991b54bd98fd75def08f047d8"
I20260926 03:04:20.015611 19297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e56d7f951354fac9b458ad6f57f13a2" candidate_uuid: "774f388b003742b98c4281f83ca5ddde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87a6306c5654a878bf61c47030e1a48"
I20260926 03:04:20.015748 19558 raft_consensus.cc:3037] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:20.015793 19297 raft_consensus.cc:3037] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:20.018075 19297 raft_consensus.cc:2454] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774f388b003742b98c4281f83ca5ddde in term 1.
I20260926 03:04:20.018064 19558 raft_consensus.cc:2454] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774f388b003742b98c4281f83ca5ddde in term 1.
I20260926 03:04:20.018481 19364 leader_election.cc:302] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 774f388b003742b98c4281f83ca5ddde, da8685a991b54bd98fd75def08f047d8; no voters:
I20260926 03:04:20.018814 19627 raft_consensus.cc:2781] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:20.019281 19627 raft_consensus.cc:686] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 1 LEADER]: Becoming Leader. State: Replica: 774f388b003742b98c4281f83ca5ddde, State: Running, Role: LEADER
I20260926 03:04:20.019750 19627 consensus_queue.cc:237] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } }
I20260926 03:04:20.024272 19155 catalog_manager.cc:5495] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde reported cstate change: term changed from 0 to 1, leader changed from <none> to 774f388b003742b98c4281f83ca5ddde (127.0.97.2). New cstate: current_term: 1 leader_uuid: "774f388b003742b98c4281f83ca5ddde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774f388b003742b98c4281f83ca5ddde" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 } health_report { overall_health: UNKNOWN } } }
W20260926 03:04:20.032161 19343 tablet.cc:2369] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:04:20.077296 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.071781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.97.2:42303)
0926 03:04:20.071995 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:20.072001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:20.072021 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:04:20.072197 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:20.072201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:20.072285 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:20.072440 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:04:20.072813 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.073597 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.074983 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.075399 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.075534 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.076175 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:04:20.076682 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:04:20.076866 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:20.077016 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 03:04:20.077062 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:04:20.085448 19558 raft_consensus.cc:1261] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 1 FOLLOWER]: Refusing update from remote peer 774f388b003742b98c4281f83ca5ddde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:20.085449 19297 raft_consensus.cc:1261] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 1 FOLLOWER]: Refusing update from remote peer 774f388b003742b98c4281f83ca5ddde: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:20.086364 19632 consensus_queue.cc:1035] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87a6306c5654a878bf61c47030e1a48" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:20.086686 19627 consensus_queue.cc:1035] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [LEADER]: Connected to new peer: Peer: permanent_uuid: "da8685a991b54bd98fd75def08f047d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:20.099282 19637 mvcc.cc:205] Tried to move back new op lower bound from 7333445058899771392 to 7333445058644733952. Current Snapshot: MvccSnapshot[applied={T|T < 7333445058899771392}]
I20260926 03:04:20.110926 19155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:20.111214 19155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:20.114550 19155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5013ffda0b5a43c195bfa8e71541a3b9: Sending DeleteTablet for 3 replicas of tablet 2e56d7f951354fac9b458ad6f57f13a2
I20260926 03:04:20.115782 19408 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e56d7f951354fac9b458ad6f57f13a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:20 UTC) from {username='slave'} at 127.0.0.1:55064
I20260926 03:04:20.115842 19277 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e56d7f951354fac9b458ad6f57f13a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:20 UTC) from {username='slave'} at 127.0.0.1:60620
I20260926 03:04:20.116289 19538 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e56d7f951354fac9b458ad6f57f13a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:20 UTC) from {username='slave'} at 127.0.0.1:37838
I20260926 03:04:20.118119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 19215
I20260926 03:04:20.126791 19647 tablet_replica.cc:330] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: stopping tablet replica
I20260926 03:04:20.128146 19647 raft_consensus.cc:2229] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:20.129580 19647 raft_consensus.cc:2258] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:20.130467 19646 tablet_replica.cc:330] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: stopping tablet replica
I20260926 03:04:20.131369 19646 raft_consensus.cc:2229] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:20.131917 19646 raft_consensus.cc:2258] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:20.133137 19647 ts_tablet_manager.cc:1905] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:20.134393 19646 ts_tablet_manager.cc:1905] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:20.141547 19646 ts_tablet_manager.cc:1918] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:20.141803 19646 log.cc:1196] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/2e56d7f951354fac9b458ad6f57f13a2
I20260926 03:04:20.142393 19646 ts_tablet_manager.cc:1939] T 2e56d7f951354fac9b458ad6f57f13a2 P c87a6306c5654a878bf61c47030e1a48: Deleting consensus metadata
I20260926 03:04:20.144420 19143 catalog_manager.cc:4841] TS c87a6306c5654a878bf61c47030e1a48 (127.0.97.1:42325): tablet 2e56d7f951354fac9b458ad6f57f13a2 (table test-table [id=14d35b227f1d4367b1f758ffe9e86e64]) successfully deleted
I20260926 03:04:20.147979 19647 ts_tablet_manager.cc:1918] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:20.148168 19647 log.cc:1196] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/2e56d7f951354fac9b458ad6f57f13a2
I20260926 03:04:20.148608 19647 ts_tablet_manager.cc:1939] T 2e56d7f951354fac9b458ad6f57f13a2 P da8685a991b54bd98fd75def08f047d8: Deleting consensus metadata
I20260926 03:04:20.150441 19143 catalog_manager.cc:4841] TS da8685a991b54bd98fd75def08f047d8 (127.0.97.3:36167): tablet 2e56d7f951354fac9b458ad6f57f13a2 (table test-table [id=14d35b227f1d4367b1f758ffe9e86e64]) successfully deleted
I20260926 03:04:20.151310 19645 tablet_replica.cc:330] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: stopping tablet replica
I20260926 03:04:20.151432 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.119649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.0.97.1:42325)
0926 03:04:20.119868 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:20.119877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:20.138895 (+ 19018us) server_negotiation.cc:407] Connection header received
0926 03:04:20.139468 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:20.139473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:20.139589 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:20.139675 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:20.140374 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.141451 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.143418 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.144050 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.144154 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.149543 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:20.149555 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:20.149571 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:20.150341 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:20.150423 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:20.150456 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:20.150542 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:20.150948 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 03:04:20.151068 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:04:20.151840 19645 raft_consensus.cc:2229] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:04:20.152613 19337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:20.152562 19645 raft_consensus.cc:2258] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:20.155827 19645 ts_tablet_manager.cc:1905] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:20.161259 19645 ts_tablet_manager.cc:1918] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:20.161443 19645 log.cc:1196] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/2e56d7f951354fac9b458ad6f57f13a2
I20260926 03:04:20.161890 19645 ts_tablet_manager.cc:1939] T 2e56d7f951354fac9b458ad6f57f13a2 P 774f388b003742b98c4281f83ca5ddde: Deleting consensus metadata
I20260926 03:04:20.163300 19140 catalog_manager.cc:4841] TS 774f388b003742b98c4281f83ca5ddde (127.0.97.2:42303): tablet 2e56d7f951354fac9b458ad6f57f13a2 (table test-table [id=14d35b227f1d4367b1f758ffe9e86e64]) successfully deleted
I20260926 03:04:20.271155 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 19215
I20260926 03:04:20.289352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 19346
I20260926 03:04:20.296953 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.290232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.0.97.2:42303)
0926 03:04:20.290471 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:20.290477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:20.290574 (+ 97us) server_negotiation.cc:407] Connection header received
0926 03:04:20.290790 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:20.290797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:20.290885 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:20.291008 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:20.291468 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.292253 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.293660 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.294229 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.294403 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.295528 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:20.295546 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:20.295560 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:20.296183 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:20.296228 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:20.296248 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:20.296368 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:20.296506 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 03:04:20.296600 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:04:20.297753 19468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:20.409788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 19346
I20260926 03:04:20.428236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 19476
I20260926 03:04:20.434602 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:20.428975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.0.97.3:36167)
0926 03:04:20.429163 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:20.429167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:20.429186 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:04:20.429320 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:20.429324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:20.429413 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:20.429496 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:20.429891 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.430588 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.431765 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:20.432131 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:20.432219 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:20.433315 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:20.433324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:20.433338 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:20.433893 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:20.433943 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:20.433961 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:20.434045 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:20.434156 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 03:04:20.434235 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:04:20.435269 19598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:20.523737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 19476
I20260926 03:04:20.540329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 19125
I20260926 03:04:20.541399 19186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:20.650656 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 19125
2026-09-26T03:04:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2954 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:20 dist-test-slave-fccw krb5kdc[19655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:20 dist-test-slave-fccw krb5kdc[19655](info): set up 2 sockets
Sep 26 03:04:20 dist-test-slave-fccw krb5kdc[19655](info): commencing operation
krb5kdc: starting...
W20260926 03:04:21.793298 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:21 dist-test-slave-fccw krb5kdc[19655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:22.039628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45025
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:22.363134 19668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:22.363430 19668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:22.363591 19668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:22.370831 19668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:22.370940 19668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:22.371053 19668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:22.371160 19668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:22.371244 19668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:22.381138 19668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:22.382498 19668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:22.383870 19668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:22.395821 19676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:22.395888 19674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:22.395856 19673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:22.398293 19668 server_base.cc:1034] running on GCE node
I20260926 03:04:22.399140 19668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:22.400522 19668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:22.401721 19668 hybrid_clock.cc:648] HybridClock initialized: now 1790391862401663 us; error 53 us; skew 500 ppm
I20260926 03:04:22.402057 19668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:22 dist-test-slave-fccw krb5kdc[19655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:22.414026 19668 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:04:22.415673 19668 webserver.cc:460] Webserver started at http://127.0.97.62:37125/ using document root <none> and password file <none>
I20260926 03:04:22.416409 19668 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:22.416514 19668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:22.416817 19668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:22.419808 19668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "ee23765d2b1640c8b0767d1b3b61335b"
format_stamp: "Formatted at 2026-09-26 03:04:22 on dist-test-slave-fccw"
server_key: "a96fdaa8fbff77bb47df22b8673901a0"
server_key_iv: "9e3dbef82d57a65fc9fae43dc43a5f5c"
server_key_version: "encryptionkey@0"
I20260926 03:04:22.420506 19668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "ee23765d2b1640c8b0767d1b3b61335b"
format_stamp: "Formatted at 2026-09-26 03:04:22 on dist-test-slave-fccw"
server_key: "a96fdaa8fbff77bb47df22b8673901a0"
server_key_iv: "9e3dbef82d57a65fc9fae43dc43a5f5c"
server_key_version: "encryptionkey@0"
I20260926 03:04:22.425683 19668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 03:04:22.428961 19683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:22.430263 19668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:22.430451 19668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "ee23765d2b1640c8b0767d1b3b61335b"
format_stamp: "Formatted at 2026-09-26 03:04:22 on dist-test-slave-fccw"
server_key: "a96fdaa8fbff77bb47df22b8673901a0"
server_key_iv: "9e3dbef82d57a65fc9fae43dc43a5f5c"
server_key_version: "encryptionkey@0"
I20260926 03:04:22.430644 19668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:22.449201 19668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:22.453425 19668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:22.453720 19668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:22.472085 19668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45025
I20260926 03:04:22.472117 19734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45025 every 8 connection(s)
I20260926 03:04:22.473453 19668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:22.477169 19735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:22.481935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 19668
I20260926 03:04:22.482121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:22.482702 388 external_mini_cluster.cc:1428] Setting key 8345f082d1d55d916df508924d132b8a
I20260926 03:04:22.488711 19735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b: Bootstrap starting.
I20260926 03:04:22.492372 19735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:22.493609 19735 log.cc:824] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:22.497095 19735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b: No bootstrap required, opened a new log
Sep 26 03:04:22 dist-test-slave-fccw krb5kdc[19655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:22.504938 19735 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee23765d2b1640c8b0767d1b3b61335b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45025 } }
I20260926 03:04:22.505461 19735 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:22.505581 19735 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee23765d2b1640c8b0767d1b3b61335b, State: Initialized, Role: FOLLOWER
I20260926 03:04:22.506181 19735 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee23765d2b1640c8b0767d1b3b61335b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45025 } }
I20260926 03:04:22.506436 19735 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:22.506480 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:22.484618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.0.97.62:45025)
0926 03:04:22.485506 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:22.485520 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:22.485566 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:04:22.487350 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:22.487373 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:22.487757 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:22.488074 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:22.489927 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:22.491350 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:22.493264 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:22.493999 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:22.494271 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:22.498797 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:22.498808 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:22.498822 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:22.498874 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:22.502107 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:22.502873 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:22.502879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:22.502885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:22.503039 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:22.503470 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:22.503474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:22.503476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:22.504021 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:22.504326 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:22.504744 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 03:04:22.505185 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":457,"threads_started":1}
I20260926 03:04:22.506585 19735 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:22.506829 19735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:22.509702 19735 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee23765d2b1640c8b0767d1b3b61335b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45025 } }
I20260926 03:04:22.510166 19735 leader_election.cc:302] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee23765d2b1640c8b0767d1b3b61335b; no voters:
I20260926 03:04:22.510674 19735 leader_election.cc:288] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:22.510850 19740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:22.511615 19740 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [term 1 LEADER]: Becoming Leader. State: Replica: ee23765d2b1640c8b0767d1b3b61335b, State: Running, Role: LEADER
I20260926 03:04:22.512203 19740 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee23765d2b1640c8b0767d1b3b61335b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45025 } }
I20260926 03:04:22.512789 19735 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:22.515627 19742 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee23765d2b1640c8b0767d1b3b61335b. Latest consensus state: current_term: 1 leader_uuid: "ee23765d2b1640c8b0767d1b3b61335b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee23765d2b1640c8b0767d1b3b61335b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45025 } } }
I20260926 03:04:22.516013 19742 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:22.517153 19749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:22.516908 19741 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee23765d2b1640c8b0767d1b3b61335b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee23765d2b1640c8b0767d1b3b61335b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45025 } } }
I20260926 03:04:22.517431 19741 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:22.523214 19749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:22.532262 19749 catalog_manager.cc:1347] Generated new cluster ID: 06501816d3734f31a97bae650e253c85
I20260926 03:04:22.532413 19749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:22.557955 19749 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:22.559381 19749 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:22.566534 19749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b: Generated new TSK 0
I20260926 03:04:22.567318 19749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260926 03:04:22.628070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45025
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:22.955564 19761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:22.955848 19761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:22.956004 19761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:22.963363 19761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:22.963476 19761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:22.963662 19761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:04:22.973457 19761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:22.974715 19761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:22.976047 19761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:22.989400 19766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:22.989516 19767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:22.989281 19769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:22.990496 19761 server_base.cc:1034] running on GCE node
I20260926 03:04:22.991140 19761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:22.992044 19761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:22.993232 19761 hybrid_clock.cc:648] HybridClock initialized: now 1790391862993182 us; error 39 us; skew 500 ppm
I20260926 03:04:22.993547 19761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:22 dist-test-slave-fccw krb5kdc[19655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:22.996515 19761 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260926 03:04:22.997949 19761 webserver.cc:460] Webserver started at http://127.0.97.1:35445/ using document root <none> and password file <none>
I20260926 03:04:22.998629 19761 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:22.998754 19761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:22.999064 19761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:23.002046 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "9ec32e02b5494c3b83b3611b0b877fa7"
format_stamp: "Formatted at 2026-09-26 03:04:22 on dist-test-slave-fccw"
server_key: "69aa4ae4a35f0378f90877cdf81264eb"
server_key_iv: "1b9239bc525076d13a369d49e0886783"
server_key_version: "encryptionkey@0"
I20260926 03:04:23.002835 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9ec32e02b5494c3b83b3611b0b877fa7"
format_stamp: "Formatted at 2026-09-26 03:04:22 on dist-test-slave-fccw"
server_key: "69aa4ae4a35f0378f90877cdf81264eb"
server_key_iv: "1b9239bc525076d13a369d49e0886783"
server_key_version: "encryptionkey@0"
I20260926 03:04:23.007887 19761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 03:04:23.011263 19776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:23.012598 19761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:04:23.012765 19761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "9ec32e02b5494c3b83b3611b0b877fa7"
format_stamp: "Formatted at 2026-09-26 03:04:22 on dist-test-slave-fccw"
server_key: "69aa4ae4a35f0378f90877cdf81264eb"
server_key_iv: "1b9239bc525076d13a369d49e0886783"
server_key_version: "encryptionkey@0"
I20260926 03:04:23.012954 19761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:23.031006 19761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:23.034778 19761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:23.035099 19761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:23.036146 19761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:23.037539 19761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:23.037636 19761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:23.037778 19761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:23.037854 19761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:23.070688 19761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44017
I20260926 03:04:23.070775 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44017 every 8 connection(s)
I20260926 03:04:23.072008 19761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:04:23.081080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 19761
I20260926 03:04:23.081269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:04:23.081773 388 external_mini_cluster.cc:1428] Setting key 438060ce89752952d3225de7d2384ec1
Sep 26 03:04:23 dist-test-slave-fccw krb5kdc[19655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:23.092121 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:23.074539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52583 (local address 127.0.97.62:45025)
0926 03:04:23.074968 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:23.074972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:23.075953 (+ 981us) server_negotiation.cc:407] Connection header received
0926 03:04:23.077728 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:23.077732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:23.077797 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:23.077897 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:23.079306 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.080008 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.081943 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.082336 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.082475 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:23.086427 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:23.086431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:23.086435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:23.086465 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:23.088931 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:23.089894 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:23.089898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:23.089900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:23.090008 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:23.090466 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:23.090469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:23.090471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:23.090698 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:23.090853 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:23.091527 (+ 674us) server_negotiation.cc:299] Negotiation successful
0926 03:04:23.091737 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":216,"threads_started":1}
I20260926 03:04:23.093453 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:23.075115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45025 (local address 127.0.97.1:52583)
0926 03:04:23.075787 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:23.075830 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:23.077487 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:23.078087 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:23.078095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:23.078608 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:23.079159 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:23.079175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.080133 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.080137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:23.081800 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:23.081816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.082579 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.082588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:23.082864 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:23.083553 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:23.083578 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:23.086220 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:23.089115 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:23.089126 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:23.089145 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:23.089775 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:23.090112 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:23.090120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:23.090126 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:23.090364 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:23.090837 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:23.090844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:23.091162 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 03:04:23.091416 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 03:04:23.091926 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":267,"threads_started":1}
I20260926 03:04:23.095361 19889 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45025
I20260926 03:04:23.095700 19889 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:23.096666 19889 heartbeater.cc:511] Master 127.0.97.62:45025 requested a full tablet report, sending...
I20260926 03:04:23.098934 19700 ts_manager.cc:194] Registered new tserver with Master: 9ec32e02b5494c3b83b3611b0b877fa7 (127.0.97.1:44017)
I20260926 03:04:23.100936 19700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52583
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260926 03:04:23.139478 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45025
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:23.458578 19895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:23.458863 19895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:23.458957 19895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:23.466028 19895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:23.466132 19895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:23.466244 19895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:04:23.476027 19895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:23.477319 19895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:23.478587 19895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:23.491895 19900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:23.491997 19901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:23.491966 19903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:23.492950 19895 server_base.cc:1034] running on GCE node
I20260926 03:04:23.493713 19895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:23.494640 19895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:23.495865 19895 hybrid_clock.cc:648] HybridClock initialized: now 1790391863495816 us; error 60 us; skew 500 ppm
I20260926 03:04:23.496172 19895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:23 dist-test-slave-fccw krb5kdc[19655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:23.499146 19895 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260926 03:04:23.500640 19895 webserver.cc:460] Webserver started at http://127.0.97.2:35219/ using document root <none> and password file <none>
I20260926 03:04:23.501384 19895 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:23.501520 19895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:23.501827 19895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:23.504581 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "132306bcfbc7457eb2f3020cdf7b2666"
format_stamp: "Formatted at 2026-09-26 03:04:23 on dist-test-slave-fccw"
server_key: "f5dd8aeee0b53214f2bff0dfa9b84ac0"
server_key_iv: "93c216b6d051e7c556d6c1ccb529b9a2"
server_key_version: "encryptionkey@0"
I20260926 03:04:23.505401 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "132306bcfbc7457eb2f3020cdf7b2666"
format_stamp: "Formatted at 2026-09-26 03:04:23 on dist-test-slave-fccw"
server_key: "f5dd8aeee0b53214f2bff0dfa9b84ac0"
server_key_iv: "93c216b6d051e7c556d6c1ccb529b9a2"
server_key_version: "encryptionkey@0"
I20260926 03:04:23.510216 19895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:04:23.513782 19910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:23.515100 19895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:23.515264 19895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "132306bcfbc7457eb2f3020cdf7b2666"
format_stamp: "Formatted at 2026-09-26 03:04:23 on dist-test-slave-fccw"
server_key: "f5dd8aeee0b53214f2bff0dfa9b84ac0"
server_key_iv: "93c216b6d051e7c556d6c1ccb529b9a2"
server_key_version: "encryptionkey@0"
I20260926 03:04:23.515453 19895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:23.530900 19895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:23.534727 19895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:23.535036 19895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:23.535941 19895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:23.537257 19895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:23.537352 19895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:23.537505 19895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:23.537577 19895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:23.569844 19895 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32801
I20260926 03:04:23.569885 20022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32801 every 8 connection(s)
I20260926 03:04:23.571094 19895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:04:23.572079 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 19895
I20260926 03:04:23.572271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:04:23.572817 388 external_mini_cluster.cc:1428] Setting key dff7a0c4ca9f183ed895daf5839260ea
Sep 26 03:04:23 dist-test-slave-fccw krb5kdc[19655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:23.596095 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:23.574040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50477 (local address 127.0.97.62:45025)
0926 03:04:23.574248 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:23.574255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:23.575348 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 03:04:23.578330 (+ 2982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:23.578338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:23.578425 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:23.578526 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:23.580533 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.581538 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.584367 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.585073 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.585205 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:23.589100 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:23.589109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:23.589117 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:23.589162 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:23.592719 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:23.593469 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:23.593477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:23.593483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:23.593635 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:23.594115 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:23.594122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:23.594129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:23.594531 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:23.594677 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:23.595359 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 03:04:23.595622 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:04:23.596685 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:23.574380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45025 (local address 127.0.97.2:50477)
0926 03:04:23.575174 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:23.575226 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:23.578028 (+ 2802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:23.578778 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:23.578792 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:23.579557 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:23.580352 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:23.580380 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.581700 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.581707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:23.583785 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:23.583802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:23.585309 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:23.585319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:23.585666 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:23.586409 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:23.586431 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:23.588831 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:23.592858 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:23.592864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:23.592878 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:23.593335 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:23.593791 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:23.593794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:23.593797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:23.594002 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:23.594667 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:23.594673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:23.594981 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 03:04:23.595207 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 03:04:23.595584 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":323,"threads_started":1}
I20260926 03:04:23.598145 20023 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45025
I20260926 03:04:23.598459 20023 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:23.599156 20023 heartbeater.cc:511] Master 127.0.97.62:45025 requested a full tablet report, sending...
I20260926 03:04:23.600603 19700 ts_manager.cc:194] Registered new tserver with Master: 132306bcfbc7457eb2f3020cdf7b2666 (127.0.97.2:32801)
I20260926 03:04:23.601512 19700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50477
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260926 03:04:23.632536 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45025
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:23.955827 20028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:23.956111 20028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:23.956204 20028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:23.963559 20028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:23.963670 20028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:23.963784 20028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:04:23.973577 20028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:23.974809 20028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:23.976092 20028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:23.989737 20034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:23.989673 20033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:23.989748 20036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:23.990568 20028 server_base.cc:1034] running on GCE node
I20260926 03:04:23.991199 20028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:23.992190 20028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:23.993403 20028 hybrid_clock.cc:648] HybridClock initialized: now 1790391863993360 us; error 40 us; skew 500 ppm
I20260926 03:04:23.993747 20028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:23 dist-test-slave-fccw krb5kdc[19655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:23.996694 20028 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260926 03:04:23.998071 20028 webserver.cc:460] Webserver started at http://127.0.97.3:33975/ using document root <none> and password file <none>
I20260926 03:04:23.998695 20028 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:23.998788 20028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:23.999032 20028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:24.001991 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e"
format_stamp: "Formatted at 2026-09-26 03:04:23 on dist-test-slave-fccw"
server_key: "0d04217ea45a7a8b2b403fa8831be6be"
server_key_iv: "374b75f0d382110899de7fbd001e8b59"
server_key_version: "encryptionkey@0"
I20260926 03:04:24.002754 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e"
format_stamp: "Formatted at 2026-09-26 03:04:23 on dist-test-slave-fccw"
server_key: "0d04217ea45a7a8b2b403fa8831be6be"
server_key_iv: "374b75f0d382110899de7fbd001e8b59"
server_key_version: "encryptionkey@0"
I20260926 03:04:24.007858 20028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:04:24.011224 20043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:24.012454 20028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:04:24.012640 20028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e"
format_stamp: "Formatted at 2026-09-26 03:04:23 on dist-test-slave-fccw"
server_key: "0d04217ea45a7a8b2b403fa8831be6be"
server_key_iv: "374b75f0d382110899de7fbd001e8b59"
server_key_version: "encryptionkey@0"
I20260926 03:04:24.012802 20028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:24.059424 20028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:24.063407 20028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:24.063706 20028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:24.064735 20028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:24.066087 20028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:24.066192 20028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:24.066354 20028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:24.066429 20028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:24.099304 20028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38517
I20260926 03:04:24.099345 20155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38517 every 8 connection(s)
I20260926 03:04:24.100558 20028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:04:24.104008 19889 heartbeater.cc:503] Master 127.0.97.62:45025 was elected leader, sending a full tablet report...
I20260926 03:04:24.105557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20028
I20260926 03:04:24.105748 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:04:24.106235 388 external_mini_cluster.cc:1428] Setting key 272e0b548e7050a1016a1582a931cc94
Sep 26 03:04:24 dist-test-slave-fccw krb5kdc[19655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:24.120698 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.103143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48291 (local address 127.0.97.62:45025)
0926 03:04:24.103644 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.103650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.104880 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 03:04:24.106968 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.106973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.107060 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.107172 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:24.108646 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.109455 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.111010 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.111386 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.111513 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.114874 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:24.114880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:24.114885 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:24.114923 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:24.117667 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.118337 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.118343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.118346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.118500 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.118922 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.118928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.118931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.119195 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:24.119368 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.120133 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.120350 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
I20260926 03:04:24.121630 20159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.103831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45025 (local address 127.0.97.3:48291)
0926 03:04:24.104709 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:24.104746 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:24.106703 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:24.107435 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:24.107444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:24.107956 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:24.108492 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.108508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.109647 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.109654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.110873 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.110883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.111592 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.111599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.111820 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.112418 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:04:24.112439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:24.114683 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:24.117788 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:24.117796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:24.117810 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:24.118201 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:24.118628 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:04:24.118633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:04:24.118637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:04:24.118806 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:04:24.119342 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:24.119349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:24.119772 (+ 423us) client_negotiation.cc:769] Sending connection context
0926 03:04:24.120043 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 03:04:24.120512 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":333,"threads_started":1}
I20260926 03:04:24.122795 20156 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45025
I20260926 03:04:24.123099 20156 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:24.123770 20156 heartbeater.cc:511] Master 127.0.97.62:45025 requested a full tablet report, sending...
I20260926 03:04:24.125101 19700 ts_manager.cc:194] Registered new tserver with Master: c20e6b28c4e3496db7cb93f6b7c7a30e (127.0.97.3:38517)
I20260926 03:04:24.125890 19700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48291
I20260926 03:04:24.132699 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:04:24.147840 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.136683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.0.97.62:45025)
0926 03:04:24.136907 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.136913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.137217 (+ 304us) server_negotiation.cc:407] Connection header received
0926 03:04:24.137334 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.137338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.137395 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.137471 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:24.138694 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.139373 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.140537 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.140908 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.141042 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.143208 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:24.143212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:24.143215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:24.143242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:24.145614 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.146199 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.146203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.146206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.146321 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.146639 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.146642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.146644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.146928 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:24.147098 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.147374 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.147527 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:04:24.151504 19700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:04:24.158219 19700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:04:24.192196 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.170496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.0.97.3:38517)
0926 03:04:24.183115 (+ 12619us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.183124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.183159 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:04:24.183249 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.183254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.183480 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.183686 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:24.184841 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.186516 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.189705 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.190994 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.191150 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.191388 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.191655 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.191815 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12409,"thread_start_us":13425,"threads_started":1}
I20260926 03:04:24.192682 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.171018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44017 (local address 127.0.0.1:47260)
0926 03:04:24.182697 (+ 11679us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:24.182722 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:24.182902 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:24.184135 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:24.184146 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:24.184168 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:24.184655 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.184665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.186685 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.186691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.189428 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.189446 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.191642 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.191650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.191866 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.191900 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:04:24.191989 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:04:24.192133 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11533,"mutex_wait_us":214,"thread_start_us":11379,"threads_started":1}
I20260926 03:04:24.192866 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.170294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38517 (local address 127.0.0.1:47242)
0926 03:04:24.171036 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:24.171097 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:24.171262 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:24.184028 (+ 12766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:24.184035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:24.184068 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:24.184596 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.184616 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.186685 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.186691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.189517 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.189537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.191181 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.191187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.191370 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.191424 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 03:04:24.191609 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 03:04:24.191877 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":88,"thread_start_us":329,"threads_started":1}
I20260926 03:04:24.192885 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.171785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.0.97.1:44017)
0926 03:04:24.181881 (+ 10096us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.181896 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.183066 (+ 1170us) server_negotiation.cc:407] Connection header received
0926 03:04:24.183186 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.183196 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.183480 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.183686 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:24.184868 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.186506 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.189642 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.191130 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.191254 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.191493 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.192372 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.192531 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9874,"thread_start_us":9695,"threads_started":1}
I20260926 03:04:24.196579 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.171469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32801 (local address 127.0.0.1:52192)
0926 03:04:24.187652 (+ 16183us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:24.187677 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:24.187781 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:24.189038 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:24.189043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:24.189067 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:24.189555 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.189566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.191452 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.191458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.194490 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.194502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.195746 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.195752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.195906 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.195941 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:04:24.196048 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:04:24.196170 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16058,"mutex_wait_us":1,"thread_start_us":14124,"threads_started":1}
I20260926 03:04:24.196626 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.172464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.0.97.2:32801)
0926 03:04:24.188416 (+ 15952us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.188427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.188457 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:04:24.188557 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.188566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.188800 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.189040 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:24.189736 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.191272 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.194633 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.195547 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.195703 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.195899 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.196107 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.196308 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15786,"thread_start_us":14965,"threads_started":1}
I20260926 03:04:24.197233 19824 tablet_service.cc:1467] Processing CreateTablet for tablet 9eaec6e1c4d149dd994d02e0d39774f7 (DEFAULT_TABLE table=test-table [id=4f2c13cb6558491192cde6d02f935f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:24.197233 20091 tablet_service.cc:1467] Processing CreateTablet for tablet 9eaec6e1c4d149dd994d02e0d39774f7 (DEFAULT_TABLE table=test-table [id=4f2c13cb6558491192cde6d02f935f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:24.198761 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eaec6e1c4d149dd994d02e0d39774f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:24.198762 20091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eaec6e1c4d149dd994d02e0d39774f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:24.200114 19958 tablet_service.cc:1467] Processing CreateTablet for tablet 9eaec6e1c4d149dd994d02e0d39774f7 (DEFAULT_TABLE table=test-table [id=4f2c13cb6558491192cde6d02f935f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:24.201200 19958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eaec6e1c4d149dd994d02e0d39774f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:24.212499 20175 tablet_bootstrap.cc:492] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Bootstrap starting.
I20260926 03:04:24.214569 20177 tablet_bootstrap.cc:492] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Bootstrap starting.
I20260926 03:04:24.216815 20175 tablet_bootstrap.cc:654] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:24.217505 20176 tablet_bootstrap.cc:492] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Bootstrap starting.
I20260926 03:04:24.218312 20175 log.cc:824] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:24.218552 20177 tablet_bootstrap.cc:654] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:24.219794 20177 log.cc:824] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:24.221158 20176 tablet_bootstrap.cc:654] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:24.221637 20175 tablet_bootstrap.cc:492] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: No bootstrap required, opened a new log
I20260926 03:04:24.221915 20175 ts_tablet_manager.cc:1397] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 03:04:24.222167 20176 log.cc:824] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:24.222844 20177 tablet_bootstrap.cc:492] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: No bootstrap required, opened a new log
I20260926 03:04:24.223091 20177 ts_tablet_manager.cc:1397] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 03:04:24.224576 20176 tablet_bootstrap.cc:492] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: No bootstrap required, opened a new log
I20260926 03:04:24.224977 20176 ts_tablet_manager.cc:1397] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260926 03:04:24.231783 20176 raft_consensus.cc:348] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.232174 20176 raft_consensus.cc:374] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:24.232280 20176 raft_consensus.cc:729] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec32e02b5494c3b83b3611b0b877fa7, State: Initialized, Role: FOLLOWER
I20260926 03:04:24.232501 20175 raft_consensus.cc:348] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.232919 20175 raft_consensus.cc:374] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:24.233037 20175 raft_consensus.cc:729] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132306bcfbc7457eb2f3020cdf7b2666, State: Initialized, Role: FOLLOWER
I20260926 03:04:24.232903 20176 consensus_queue.cc:260] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.233588 20175 consensus_queue.cc:260] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.233494 20177 raft_consensus.cc:348] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.233932 20177 raft_consensus.cc:374] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:24.234099 20177 raft_consensus.cc:729] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20e6b28c4e3496db7cb93f6b7c7a30e, State: Initialized, Role: FOLLOWER
I20260926 03:04:24.234925 20177 consensus_queue.cc:260] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.237911 20023 heartbeater.cc:503] Master 127.0.97.62:45025 was elected leader, sending a full tablet report...
I20260926 03:04:24.239176 20175 ts_tablet_manager.cc:1428] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260926 03:04:24.240737 20176 ts_tablet_manager.cc:1428] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260926 03:04:24.242146 20156 heartbeater.cc:503] Master 127.0.97.62:45025 was elected leader, sending a full tablet report...
I20260926 03:04:24.242723 20177 ts_tablet_manager.cc:1428] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
W20260926 03:04:24.323599 20024 tablet.cc:2369] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:04:24.324440 19890 tablet.cc:2369] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:04:24.352324 20157 tablet.cc:2369] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:04:24.436424 20182 raft_consensus.cc:482] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:04:24.436729 20182 raft_consensus.cc:504] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.438405 20182 leader_election.cc:288] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c20e6b28c4e3496db7cb93f6b7c7a30e (127.0.97.3:38517), 9ec32e02b5494c3b83b3611b0b877fa7 (127.0.97.1:44017)
I20260926 03:04:24.446306 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.438688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40159 (local address 127.0.97.3:38517)
0926 03:04:24.438916 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.438926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.439628 (+ 702us) server_negotiation.cc:407] Connection header received
0926 03:04:24.440034 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.440038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.440100 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.440211 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:24.440811 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.441924 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.444326 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.445511 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.445644 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.445745 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.445975 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.446122 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 03:04:24.446480 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.438616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38517 (local address 127.0.97.2:40159)
0926 03:04:24.439506 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:24.439530 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:24.439850 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:24.440228 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:24.440232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:24.440258 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:24.440644 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.440656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.442055 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.442062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.444189 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.444202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.445694 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.445698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.445803 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.445827 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:04:24.445938 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 03:04:24.446096 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":508,"threads_started":1}
I20260926 03:04:24.446795 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.438844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40983 (local address 127.0.97.1:44017)
0926 03:04:24.439218 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.439226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.440119 (+ 893us) server_negotiation.cc:407] Connection header received
0926 03:04:24.440868 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.440896 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.441022 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.441128 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:24.441717 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.442384 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.444566 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.445813 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.445947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.446041 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.446404 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.446538 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260926 03:04:24.446825 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.438784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44017 (local address 127.0.97.2:40983)
0926 03:04:24.440025 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:24.440038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:24.440158 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:24.441164 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:24.441169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:24.441195 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:24.441575 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.441586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.442516 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.442522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.444433 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:24.444442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.446012 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.446017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:24.446208 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.446246 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:04:24.446350 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 03:04:24.446485 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":1,"thread_start_us":766,"threads_started":1}
I20260926 03:04:24.447680 20111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eaec6e1c4d149dd994d02e0d39774f7" candidate_uuid: "132306bcfbc7457eb2f3020cdf7b2666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" is_pre_election: true
I20260926 03:04:24.448024 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eaec6e1c4d149dd994d02e0d39774f7" candidate_uuid: "132306bcfbc7457eb2f3020cdf7b2666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" is_pre_election: true
I20260926 03:04:24.448257 20111 raft_consensus.cc:2454] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 132306bcfbc7457eb2f3020cdf7b2666 in term 0.
I20260926 03:04:24.448561 19844 raft_consensus.cc:2454] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 132306bcfbc7457eb2f3020cdf7b2666 in term 0.
I20260926 03:04:24.448971 19913 leader_election.cc:302] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132306bcfbc7457eb2f3020cdf7b2666, c20e6b28c4e3496db7cb93f6b7c7a30e; no voters:
I20260926 03:04:24.449429 20182 raft_consensus.cc:2781] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:04:24.449574 20182 raft_consensus.cc:482] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:04:24.449681 20182 raft_consensus.cc:3037] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:24.451762 20182 raft_consensus.cc:504] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.452344 20182 leader_election.cc:288] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [CANDIDATE]: Term 1 election: Requested vote from peers c20e6b28c4e3496db7cb93f6b7c7a30e (127.0.97.3:38517), 9ec32e02b5494c3b83b3611b0b877fa7 (127.0.97.1:44017)
I20260926 03:04:24.452698 20111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eaec6e1c4d149dd994d02e0d39774f7" candidate_uuid: "132306bcfbc7457eb2f3020cdf7b2666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e"
I20260926 03:04:24.452767 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eaec6e1c4d149dd994d02e0d39774f7" candidate_uuid: "132306bcfbc7457eb2f3020cdf7b2666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec32e02b5494c3b83b3611b0b877fa7"
I20260926 03:04:24.452915 20111 raft_consensus.cc:3037] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:24.452980 19844 raft_consensus.cc:3037] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:24.455128 20111 raft_consensus.cc:2454] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 132306bcfbc7457eb2f3020cdf7b2666 in term 1.
I20260926 03:04:24.455142 19844 raft_consensus.cc:2454] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 132306bcfbc7457eb2f3020cdf7b2666 in term 1.
I20260926 03:04:24.455530 19913 leader_election.cc:302] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132306bcfbc7457eb2f3020cdf7b2666, c20e6b28c4e3496db7cb93f6b7c7a30e; no voters:
I20260926 03:04:24.455814 20182 raft_consensus.cc:2781] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:24.456248 20182 raft_consensus.cc:686] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 1 LEADER]: Becoming Leader. State: Replica: 132306bcfbc7457eb2f3020cdf7b2666, State: Running, Role: LEADER
I20260926 03:04:24.456710 20182 consensus_queue.cc:237] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } }
I20260926 03:04:24.461377 19698 catalog_manager.cc:5495] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 132306bcfbc7457eb2f3020cdf7b2666 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "132306bcfbc7457eb2f3020cdf7b2666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "132306bcfbc7457eb2f3020cdf7b2666" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 } health_report { overall_health: UNKNOWN } } }
I20260926 03:04:24.490615 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.485701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.97.2:32801)
0926 03:04:24.485875 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.485880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.485895 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:04:24.485996 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.485999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.486145 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.486262 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:04:24.486590 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.487320 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.488803 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.489201 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.489287 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.489721 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:04:24.490193 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:04:24.490302 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.490369 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.490427 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:04:24.498301 19844 raft_consensus.cc:1261] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 1 FOLLOWER]: Refusing update from remote peer 132306bcfbc7457eb2f3020cdf7b2666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:24.498306 20111 raft_consensus.cc:1261] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 1 FOLLOWER]: Refusing update from remote peer 132306bcfbc7457eb2f3020cdf7b2666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:24.499238 20182 consensus_queue.cc:1035] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec32e02b5494c3b83b3611b0b877fa7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:24.499511 20186 consensus_queue.cc:1035] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20e6b28c4e3496db7cb93f6b7c7a30e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:24.510643 20189 mvcc.cc:205] Tried to move back new op lower bound from 7333445076975611904 to 7333445076818288640. Current Snapshot: MvccSnapshot[applied={T|T < 7333445076975611904}]
I20260926 03:04:24.514994 20191 mvcc.cc:205] Tried to move back new op lower bound from 7333445076975611904 to 7333445076818288640. Current Snapshot: MvccSnapshot[applied={T|T < 7333445076975611904}]
I20260926 03:04:24.525453 19700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:24.525758 19700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:24.529361 19700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee23765d2b1640c8b0767d1b3b61335b: Sending DeleteTablet for 3 replicas of tablet 9eaec6e1c4d149dd994d02e0d39774f7
I20260926 03:04:24.530422 20091 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eaec6e1c4d149dd994d02e0d39774f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47242
I20260926 03:04:24.530519 19958 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eaec6e1c4d149dd994d02e0d39774f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52192
I20260926 03:04:24.530576 19824 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eaec6e1c4d149dd994d02e0d39774f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47260
I20260926 03:04:24.532045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 19761
I20260926 03:04:24.545838 20199 tablet_replica.cc:330] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: stopping tablet replica
I20260926 03:04:24.546353 20199 raft_consensus.cc:2229] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:24.546919 20199 raft_consensus.cc:2258] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:24.547717 20200 tablet_replica.cc:330] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: stopping tablet replica
I20260926 03:04:24.548076 20200 raft_consensus.cc:2229] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:04:24.548714 20200 raft_consensus.cc:2258] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:24.549481 20201 tablet_replica.cc:330] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: stopping tablet replica
I20260926 03:04:24.549937 20201 raft_consensus.cc:2229] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:24.550424 20201 raft_consensus.cc:2258] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:24.551566 20200 ts_tablet_manager.cc:1905] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:24.551569 20199 ts_tablet_manager.cc:1905] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:24.552327 20201 ts_tablet_manager.cc:1905] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 03:04:24 dist-test-slave-fccw krb5kdc[19655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260926 03:04:24.558562 20200 ts_tablet_manager.cc:1918] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:24.558806 20200 log.cc:1196] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/9eaec6e1c4d149dd994d02e0d39774f7
I20260926 03:04:24.558815 20199 ts_tablet_manager.cc:1918] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:24.559013 20199 log.cc:1196] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/9eaec6e1c4d149dd994d02e0d39774f7
I20260926 03:04:24.559124 20201 ts_tablet_manager.cc:1918] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:24.559286 20201 log.cc:1196] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/9eaec6e1c4d149dd994d02e0d39774f7
I20260926 03:04:24.559564 20199 ts_tablet_manager.cc:1939] T 9eaec6e1c4d149dd994d02e0d39774f7 P c20e6b28c4e3496db7cb93f6b7c7a30e: Deleting consensus metadata
I20260926 03:04:24.559760 20200 ts_tablet_manager.cc:1939] T 9eaec6e1c4d149dd994d02e0d39774f7 P 132306bcfbc7457eb2f3020cdf7b2666: Deleting consensus metadata
I20260926 03:04:24.559793 20201 ts_tablet_manager.cc:1939] T 9eaec6e1c4d149dd994d02e0d39774f7 P 9ec32e02b5494c3b83b3611b0b877fa7: Deleting consensus metadata
I20260926 03:04:24.561421 19684 catalog_manager.cc:4841] TS 9ec32e02b5494c3b83b3611b0b877fa7 (127.0.97.1:44017): tablet 9eaec6e1c4d149dd994d02e0d39774f7 (table test-table [id=4f2c13cb6558491192cde6d02f935f02]) successfully deleted
I20260926 03:04:24.561779 19686 catalog_manager.cc:4841] TS 132306bcfbc7457eb2f3020cdf7b2666 (127.0.97.2:32801): tablet 9eaec6e1c4d149dd994d02e0d39774f7 (table test-table [id=4f2c13cb6558491192cde6d02f935f02]) successfully deleted
I20260926 03:04:24.563807 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.533507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.0.97.1:44017)
0926 03:04:24.533747 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.533756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.544642 (+ 10886us) server_negotiation.cc:407] Connection header received
0926 03:04:24.545198 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.545209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.545330 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.545415 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:24.547306 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.548778 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.550702 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.551356 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.551473 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.556648 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:24.556661 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:24.556677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:24.556714 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:24.560751 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.561712 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.561719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.561724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.561875 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.562305 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.562310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.562313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.562649 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:24.562950 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.563250 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.563453 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:04:24.564213 19686 catalog_manager.cc:4841] TS c20e6b28c4e3496db7cb93f6b7c7a30e (127.0.97.3:38517): tablet 9eaec6e1c4d149dd994d02e0d39774f7 (table test-table [id=4f2c13cb6558491192cde6d02f935f02]) successfully deleted
I20260926 03:04:24.564546 19884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:24.682139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 19761
I20260926 03:04:24.700429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 19895
Sep 26 03:04:24 dist-test-slave-fccw krb5kdc[19655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260926 03:04:24.716801 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.701279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.0.97.2:32801)
0926 03:04:24.701475 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.701482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.701578 (+ 96us) server_negotiation.cc:407] Connection header received
0926 03:04:24.701768 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.701773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.701852 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.701967 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:24.703494 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.704342 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.705721 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.706161 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.706299 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.710651 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:24.710663 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:24.710679 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:24.710711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:24.713864 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.714592 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.714600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.714602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.714735 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.715123 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.715130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.715133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.715559 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:24.715907 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.716166 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.716404 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 03:04:24.717542 20018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:24.827898 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 19895
I20260926 03:04:24.846299 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20028
Sep 26 03:04:24 dist-test-slave-fccw krb5kdc[19655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260926 03:04:24.860200 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:24.846968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.0.97.3:38517)
0926 03:04:24.847125 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:24.847129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:24.847163 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:04:24.847343 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:24.847347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:24.847410 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:24.847492 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:24.848706 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.849492 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.850753 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:24.851104 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:24.851188 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:24.855417 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:24.855428 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:24.855442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:24.855475 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:24.857935 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.858521 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.858527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.858529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.858654 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:24.858954 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:24.858957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:24.858958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:24.859267 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:24.859487 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:24.859679 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 03:04:24.859822 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:04:24.860795 20151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:24.949065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20028
I20260926 03:04:24.966105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 19668
I20260926 03:04:24.967175 19730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:25.077265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 19668
2026-09-26T03:04:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4430 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T03:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:25Z Disabled control of system clock
I20260926 03:04:25.152292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34195
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:04:25.474785 20210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:25.475078 20210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:25.475256 20210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:25.482560 20210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:25.482667 20210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:25.482707 20210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:25.482787 20210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:25.482852 20210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:25.492094 20210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34195
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:25.493482 20210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:25.495252 20210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:25.507287 20215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:25.507366 20218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:25.507416 20216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:25.510358 20210 server_base.cc:1034] running on GCE node
I20260926 03:04:25.511128 20210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:25.512496 20210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:25.513700 20210 hybrid_clock.cc:648] HybridClock initialized: now 1790391865513654 us; error 40 us; skew 500 ppm
I20260926 03:04:25.514012 20210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:25.516192 20210 webserver.cc:460] Webserver started at http://127.0.97.62:36221/ using document root <none> and password file <none>
I20260926 03:04:25.516837 20210 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:25.516924 20210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:25.517261 20210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:25.520143 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "0e942a1eda464bcda857602f3a85c4b7"
format_stamp: "Formatted at 2026-09-26 03:04:25 on dist-test-slave-fccw"
server_key: "6782bcccdadda8388f84b8af08c497eb"
server_key_iv: "b3e85e5c3d40ed60cee4c5918be41fd5"
server_key_version: "encryptionkey@0"
I20260926 03:04:25.520844 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e942a1eda464bcda857602f3a85c4b7"
format_stamp: "Formatted at 2026-09-26 03:04:25 on dist-test-slave-fccw"
server_key: "6782bcccdadda8388f84b8af08c497eb"
server_key_iv: "b3e85e5c3d40ed60cee4c5918be41fd5"
server_key_version: "encryptionkey@0"
I20260926 03:04:25.525858 20210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 03:04:25.529148 20224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:25.530393 20210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:04:25.530529 20210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "0e942a1eda464bcda857602f3a85c4b7"
format_stamp: "Formatted at 2026-09-26 03:04:25 on dist-test-slave-fccw"
server_key: "6782bcccdadda8388f84b8af08c497eb"
server_key_iv: "b3e85e5c3d40ed60cee4c5918be41fd5"
server_key_version: "encryptionkey@0"
I20260926 03:04:25.530694 20210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:25.551589 20210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:25.552389 20210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:25.552639 20210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:25.570916 20210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34195
I20260926 03:04:25.570935 20275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34195 every 8 connection(s)
I20260926 03:04:25.572304 20210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:25.572829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20210
I20260926 03:04:25.573031 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:25.573678 388 external_mini_cluster.cc:1428] Setting key 4da896e6f0f78212a5ae928522eebdc1
I20260926 03:04:25.576854 20276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:25.586679 20279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:25.575644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.97.62:34195)
0926 03:04:25.576565 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:25.576576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:25.576615 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:04:25.578361 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:25.578382 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:25.578813 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:25.579205 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:25.579766 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:25.581113 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:25.582445 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:25.583066 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:25.583281 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:25.584432 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:25.584442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:25.584454 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:25.585046 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:25.585134 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:25.585157 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:25.585280 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:25.585502 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 03:04:25.585794 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":321,"threads_started":1}
I20260926 03:04:25.587534 20276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7: Bootstrap starting.
I20260926 03:04:25.590927 20276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:25.591845 20276 log.cc:824] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:25.594579 20276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7: No bootstrap required, opened a new log
I20260926 03:04:25.601589 20276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e942a1eda464bcda857602f3a85c4b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34195 } }
I20260926 03:04:25.601960 20276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:25.602084 20276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e942a1eda464bcda857602f3a85c4b7, State: Initialized, Role: FOLLOWER
I20260926 03:04:25.602687 20276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e942a1eda464bcda857602f3a85c4b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34195 } }
I20260926 03:04:25.602941 20276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:25.603091 20276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:25.603266 20276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:25.605559 20276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e942a1eda464bcda857602f3a85c4b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34195 } }
I20260926 03:04:25.606122 20276 leader_election.cc:302] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e942a1eda464bcda857602f3a85c4b7; no voters:
I20260926 03:04:25.606678 20276 leader_election.cc:288] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:25.606865 20281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:25.607601 20281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 0e942a1eda464bcda857602f3a85c4b7, State: Running, Role: LEADER
I20260926 03:04:25.608131 20281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e942a1eda464bcda857602f3a85c4b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34195 } }
I20260926 03:04:25.608762 20276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:25.610752 20282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e942a1eda464bcda857602f3a85c4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e942a1eda464bcda857602f3a85c4b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34195 } } }
I20260926 03:04:25.612479 20282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:25.613350 20283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e942a1eda464bcda857602f3a85c4b7. Latest consensus state: current_term: 1 leader_uuid: "0e942a1eda464bcda857602f3a85c4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e942a1eda464bcda857602f3a85c4b7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34195 } } }
I20260926 03:04:25.613814 20283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:25.614245 20290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:25.619997 20290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:25.628016 20290 catalog_manager.cc:1347] Generated new cluster ID: 9ae61e90d60f44e3a6ab4d93b4354d79
I20260926 03:04:25.628124 20290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:25.641476 20290 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:25.642731 20290 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:25.649783 20290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7: Generated new TSK 0
I20260926 03:04:25.650597 20290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:25.662344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34195
--builtin_ntp_servers=127.0.97.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 03:04:25.992718 20300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:25.993037 20300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:25.993193 20300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:26.004482 20300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:26.004689 20300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:26.004937 20300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260926 03:04:26.020247 20300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:26.022141 20300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:26.024829 20300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:26.042598 20305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:26.043284 20306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:26.043851 20300 server_base.cc:1034] running on GCE node
W20260926 03:04:26.044166 20308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:26.045011 20300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:26.046195 20300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:26.047415 20300 hybrid_clock.cc:648] HybridClock initialized: now 1790391866047379 us; error 27 us; skew 500 ppm
I20260926 03:04:26.047693 20300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:26.049991 20300 webserver.cc:460] Webserver started at http://127.0.97.1:38029/ using document root <none> and password file <none>
I20260926 03:04:26.050681 20300 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:26.050798 20300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:26.051177 20300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:26.054086 20300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data/instance:
uuid: "5025bbd064f54047a1a282493c2651fd"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "2cc35437c6b54c4a3871f924c1fa5c9a"
server_key_iv: "3957c905173a9c1ed6c7000409cc4361"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.054854 20300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5025bbd064f54047a1a282493c2651fd"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "2cc35437c6b54c4a3871f924c1fa5c9a"
server_key_iv: "3957c905173a9c1ed6c7000409cc4361"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.059841 20300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:04:26.063169 20314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:26.064499 20300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 03:04:26.064663 20300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal
uuid: "5025bbd064f54047a1a282493c2651fd"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "2cc35437c6b54c4a3871f924c1fa5c9a"
server_key_iv: "3957c905173a9c1ed6c7000409cc4361"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.064882 20300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:26.099663 20300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:26.100591 20300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:26.100868 20300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:26.101868 20300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:26.103341 20300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:26.103475 20300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:26.103609 20300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:26.103698 20300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:26.136718 20300 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39767
I20260926 03:04:26.136772 20426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39767 every 8 connection(s)
I20260926 03:04:26.138115 20300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/data/info.pb
I20260926 03:04:26.139433 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20300
I20260926 03:04:26.139602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal/instance
I20260926 03:04:26.140044 388 external_mini_cluster.cc:1428] Setting key 06e97e1dec9f6660125bd30eebd076b0
I20260926 03:04:26.143220 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34195
--builtin_ntp_servers=127.0.97.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:04:26.153165 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:26.140642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50793 (local address 127.0.97.62:34195)
0926 03:04:26.141136 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:26.141140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:26.142596 (+ 1456us) server_negotiation.cc:407] Connection header received
0926 03:04:26.145414 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:26.145418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:26.145480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:26.145570 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:26.146920 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.147617 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.150125 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.150516 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.150610 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:26.152269 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:26.152274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:26.152277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:26.152300 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:26.152343 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:26.152356 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:26.152490 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:26.152693 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 03:04:26.152788 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260926 03:04:26.154124 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:26.141394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34195 (local address 127.0.97.1:50793)
0926 03:04:26.142434 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:26.142487 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:26.145135 (+ 2648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:26.145847 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:26.145861 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:26.145978 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:26.146760 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:26.146785 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.147779 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.147787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:26.149983 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:26.150005 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.150735 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.150764 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:26.151081 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:26.151943 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:26.151945 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:26.151983 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:26.152058 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:26.152466 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:26.152486 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:26.152493 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:04:26.152684 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 03:04:26.153084 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":283,"threads_started":1}
I20260926 03:04:26.155627 20427 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34195
I20260926 03:04:26.155936 20427 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:26.156673 20427 heartbeater.cc:511] Master 127.0.97.62:34195 requested a full tablet report, sending...
I20260926 03:04:26.158670 20241 ts_manager.cc:194] Registered new tserver with Master: 5025bbd064f54047a1a282493c2651fd (127.0.97.1:39767)
I20260926 03:04:26.160102 20241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:50793
W20260926 03:04:26.465404 20431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:26.465703 20431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:26.465867 20431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:26.473032 20431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:26.473174 20431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:26.473371 20431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260926 03:04:26.482431 20431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:26.483673 20431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:26.485483 20431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:26.498230 20436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:26.498265 20439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:26.498196 20437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:26.499182 20431 server_base.cc:1034] running on GCE node
I20260926 03:04:26.499789 20431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:26.500648 20431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:26.501840 20431 hybrid_clock.cc:648] HybridClock initialized: now 1790391866501787 us; error 53 us; skew 500 ppm
I20260926 03:04:26.502147 20431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:26.504361 20431 webserver.cc:460] Webserver started at http://127.0.97.2:44561/ using document root <none> and password file <none>
I20260926 03:04:26.505079 20431 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:26.505193 20431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:26.505519 20431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:26.508443 20431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data/instance:
uuid: "e2693836d49a4d61b967ea0762f04594"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "47402559894aa797737757062205f291"
server_key_iv: "3cabb9b4240792e9ce625d71eb7078c3"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.509299 20431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e2693836d49a4d61b967ea0762f04594"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "47402559894aa797737757062205f291"
server_key_iv: "3cabb9b4240792e9ce625d71eb7078c3"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.514263 20431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:04:26.517704 20445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:26.518986 20431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:26.519162 20431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal
uuid: "e2693836d49a4d61b967ea0762f04594"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "47402559894aa797737757062205f291"
server_key_iv: "3cabb9b4240792e9ce625d71eb7078c3"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.519384 20431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:26.553131 20431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:26.553995 20431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:26.554279 20431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:26.555265 20431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:26.556849 20431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:26.556938 20431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:26.557119 20431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:26.557204 20431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:26.591082 20431 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41951
I20260926 03:04:26.591157 20557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41951 every 8 connection(s)
I20260926 03:04:26.592429 20431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/data/info.pb
I20260926 03:04:26.599897 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20431
I20260926 03:04:26.600059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal/instance
I20260926 03:04:26.600493 388 external_mini_cluster.cc:1428] Setting key 6d6a0f73a3608dbd595d7d2c082fd8bb
I20260926 03:04:26.603530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34195
--builtin_ntp_servers=127.0.97.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 03:04:26.605147 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:26.595035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43693 (local address 127.0.97.62:34195)
0926 03:04:26.595207 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:26.595211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:26.596578 (+ 1367us) server_negotiation.cc:407] Connection header received
0926 03:04:26.598349 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:26.598361 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:26.598430 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:26.598516 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:26.599479 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.600279 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.602093 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.602633 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.602776 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:26.604200 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:26.604209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:26.604217 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:26.604243 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:26.604287 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:26.604306 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:26.604411 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:26.604609 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 03:04:26.604719 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:04:26.605985 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:26.595588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34195 (local address 127.0.97.2:43693)
0926 03:04:26.596429 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:26.596462 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:26.598108 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:26.598702 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:26.598710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:26.598791 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:26.599340 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:26.599356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.600423 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.600430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:26.601943 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:26.601955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:26.602855 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:26.602860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:26.603066 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:26.603877 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:26.603880 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:26.603917 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:26.604014 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:26.604413 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:26.604420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:26.604426 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:04:26.604561 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 03:04:26.604855 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":254,"threads_started":1}
I20260926 03:04:26.607440 20558 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34195
I20260926 03:04:26.607741 20558 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:26.608454 20558 heartbeater.cc:511] Master 127.0.97.62:34195 requested a full tablet report, sending...
I20260926 03:04:26.609949 20241 ts_manager.cc:194] Registered new tserver with Master: e2693836d49a4d61b967ea0762f04594 (127.0.97.2:41951)
I20260926 03:04:26.610737 20241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:43693
W20260926 03:04:26.929983 20561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:26.930299 20561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:26.930469 20561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:26.937969 20561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:26.938107 20561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:26.938313 20561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260926 03:04:26.948772 20561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:26.950073 20561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:26.951859 20561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:26.965303 20566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:26.965247 20567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:26.965391 20569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:26.966214 20561 server_base.cc:1034] running on GCE node
I20260926 03:04:26.966832 20561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:26.967871 20561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:26.969105 20561 hybrid_clock.cc:648] HybridClock initialized: now 1790391866969038 us; error 62 us; skew 500 ppm
I20260926 03:04:26.969515 20561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 03:04:26.971719 20561 webserver.cc:460] Webserver started at http://127.0.97.3:39309/ using document root <none> and password file <none>
I20260926 03:04:26.972440 20561 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:26.972563 20561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:26.972878 20561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:26.975879 20561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data/instance:
uuid: "89627965962e464e864b1d4144001d29"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "fc8139aade3ab80c7c82902f667fffb4"
server_key_iv: "f07fdbc82ca4423d593173a209488827"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.976689 20561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal/instance:
uuid: "89627965962e464e864b1d4144001d29"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "fc8139aade3ab80c7c82902f667fffb4"
server_key_iv: "f07fdbc82ca4423d593173a209488827"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.981827 20561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:04:26.985193 20575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:26.986470 20561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:04:26.986652 20561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal
uuid: "89627965962e464e864b1d4144001d29"
format_stamp: "Formatted at 2026-09-26 03:04:26 on dist-test-slave-fccw"
server_key: "fc8139aade3ab80c7c82902f667fffb4"
server_key_iv: "f07fdbc82ca4423d593173a209488827"
server_key_version: "encryptionkey@0"
I20260926 03:04:26.986871 20561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:27.009677 20561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:27.010566 20561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:27.010855 20561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:27.011821 20561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:04:27.013635 20561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:04:27.013752 20561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:27.013902 20561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:04:27.014002 20561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:27.046629 20561 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45043
I20260926 03:04:27.046720 20687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45043 every 8 connection(s)
I20260926 03:04:27.047956 20561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/data/info.pb
I20260926 03:04:27.048892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20561
I20260926 03:04:27.049081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal/instance
I20260926 03:04:27.049624 388 external_mini_cluster.cc:1428] Setting key d6ab1380f410922656a8ba054c55d59e
I20260926 03:04:27.060401 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.050782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60957 (local address 127.0.97.62:34195)
0926 03:04:27.051066 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.051072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.052698 (+ 1626us) server_negotiation.cc:407] Connection header received
0926 03:04:27.054571 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.054576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.054641 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.054721 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:27.055747 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.056504 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.058035 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.058435 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.058537 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.059734 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:27.059739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:27.059742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:27.059761 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:27.059792 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:27.059803 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:27.059883 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.060083 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.060156 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260926 03:04:27.061435 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.051478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34195 (local address 127.0.97.3:60957)
0926 03:04:27.052535 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.052585 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:27.054333 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:27.054881 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:27.054888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:27.054968 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:04:27.055597 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.055613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.056620 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.056623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.057900 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.057911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.058626 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.058630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.058824 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.059454 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:04:27.059457 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:04:27.059494 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:04:27.059570 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:04:27.059896 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:04:27.059903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:04:27.059908 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:04:27.060063 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 03:04:27.060342 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":336,"threads_started":1}
I20260926 03:04:27.062830 20688 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34195
I20260926 03:04:27.063112 20688 heartbeater.cc:464] Registering TS with master...
I20260926 03:04:27.063822 20688 heartbeater.cc:511] Master 127.0.97.62:34195 requested a full tablet report, sending...
I20260926 03:04:27.065400 20241 ts_manager.cc:194] Registered new tserver with Master: 89627965962e464e864b1d4144001d29 (127.0.97.3:45043)
I20260926 03:04:27.066131 20241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:60957
I20260926 03:04:27.076857 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:04:27.086784 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.080951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.0.97.62:34195)
0926 03:04:27.081202 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.081209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.081543 (+ 334us) server_negotiation.cc:407] Connection header received
0926 03:04:27.081726 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.081731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.081812 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.081921 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:27.082438 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.083127 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.084472 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.084879 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.085044 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.086105 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:27.086110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:27.086113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:27.086130 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:27.086159 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:27.086172 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:27.086266 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.086414 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.086501 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:04:27.090389 20241 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:04:27.098228 20241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:04:27.134902 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.111045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.0.97.3:45043)
0926 03:04:27.123540 (+ 12495us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.123552 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.123593 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:04:27.123711 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.123722 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.124060 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.125299 (+ 1239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:27.126191 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.128517 (+ 2326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.131579 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.133051 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.133199 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.133478 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.134268 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.134463 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12268,"thread_start_us":11788,"threads_started":1}
I20260926 03:04:27.135686 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.110766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45043 (local address 127.0.0.1:39464)
0926 03:04:27.111922 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.111960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:27.112132 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:27.125351 (+ 13219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:27.125363 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:27.125408 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:27.125907 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.125924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.128649 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.128657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.131349 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.131371 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.133777 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.133783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.133995 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.134042 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 03:04:27.135121 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0926 03:04:27.135324 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":1,"thread_start_us":20219,"threads_started":1}
I20260926 03:04:27.137899 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.111096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37322)
0926 03:04:27.129494 (+ 18398us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.129513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:27.129635 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:27.130737 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:27.130745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:27.130787 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:27.131313 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.131323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.132956 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.132964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.134961 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.134973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.137051 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.137060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.137263 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.137302 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:04:27.137412 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 03:04:27.137558 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18246,"mutex_wait_us":3,"thread_start_us":817,"threads_started":1}
I20260926 03:04:27.138087 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.111754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.0.97.1:39767)
0926 03:04:27.130128 (+ 18374us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.130137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.130168 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:04:27.130261 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.130269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.130518 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.130709 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:27.131557 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.132830 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.135200 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.136542 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.136662 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.136869 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.137482 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.137669 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18193,"thread_start_us":18093,"threads_started":1}
I20260926 03:04:27.139799 20623 tablet_service.cc:1467] Processing CreateTablet for tablet a4b0434132134124b4a904205653d400 (DEFAULT_TABLE table=test-table [id=a8357c6e0b6a4cec967ab9356616b098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:27.141374 20623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0434132134124b4a904205653d400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:27.141811 20362 tablet_service.cc:1467] Processing CreateTablet for tablet a4b0434132134124b4a904205653d400 (DEFAULT_TABLE table=test-table [id=a8357c6e0b6a4cec967ab9356616b098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:27.143227 20362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0434132134124b4a904205653d400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:27.144845 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.112836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41951 (local address 127.0.0.1:57330)
0926 03:04:27.131320 (+ 18484us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.131348 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:27.131483 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:27.132121 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:27.132126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:27.132151 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:27.132475 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.132486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.134286 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.134291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.136455 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.136469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.144077 (+ 7608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.144083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.144251 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.144286 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:04:27.144416 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:04:27.144547 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18330,"thread_start_us":16343,"threads_started":1}
I20260926 03:04:27.144950 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.113211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.0.97.2:41951)
0926 03:04:27.127800 (+ 14589us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.127811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.131464 (+ 3653us) server_negotiation.cc:407] Connection header received
0926 03:04:27.131675 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.131683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.131912 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.132111 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:27.132637 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.134130 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.136660 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.143868 (+ 7208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.144044 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.144281 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.144458 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.144619 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14384,"thread_start_us":14244,"threads_started":1}
I20260926 03:04:27.147934 20493 tablet_service.cc:1467] Processing CreateTablet for tablet a4b0434132134124b4a904205653d400 (DEFAULT_TABLE table=test-table [id=a8357c6e0b6a4cec967ab9356616b098]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:04:27.149144 20493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b0434132134124b4a904205653d400. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:27.156127 20706 tablet_bootstrap.cc:492] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Bootstrap starting.
I20260926 03:04:27.158277 20707 tablet_bootstrap.cc:492] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: Bootstrap starting.
I20260926 03:04:27.160770 20706 tablet_bootstrap.cc:654] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:27.161582 20707 tablet_bootstrap.cc:654] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:27.162518 20707 log.cc:824] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:27.162636 20706 log.cc:824] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:27.162919 20427 heartbeater.cc:503] Master 127.0.97.62:34195 was elected leader, sending a full tablet report...
I20260926 03:04:27.167371 20707 tablet_bootstrap.cc:492] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: No bootstrap required, opened a new log
I20260926 03:04:27.167656 20707 ts_tablet_manager.cc:1397] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 03:04:27.168100 20708 tablet_bootstrap.cc:492] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Bootstrap starting.
I20260926 03:04:27.170815 20706 tablet_bootstrap.cc:492] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: No bootstrap required, opened a new log
I20260926 03:04:27.171049 20708 tablet_bootstrap.cc:654] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:27.171095 20706 ts_tablet_manager.cc:1397] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260926 03:04:27.172075 20708 log.cc:824] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:27.175199 20708 tablet_bootstrap.cc:492] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: No bootstrap required, opened a new log
I20260926 03:04:27.175493 20708 ts_tablet_manager.cc:1397] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 03:04:27.178045 20707 raft_consensus.cc:348] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.178515 20707 raft_consensus.cc:374] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:27.178622 20707 raft_consensus.cc:729] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5025bbd064f54047a1a282493c2651fd, State: Initialized, Role: FOLLOWER
I20260926 03:04:27.179255 20707 consensus_queue.cc:260] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.181113 20707 ts_tablet_manager.cc:1428] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260926 03:04:27.181658 20706 raft_consensus.cc:348] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.182031 20706 raft_consensus.cc:374] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:27.182132 20706 raft_consensus.cc:729] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89627965962e464e864b1d4144001d29, State: Initialized, Role: FOLLOWER
I20260926 03:04:27.182698 20706 consensus_queue.cc:260] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.183745 20688 heartbeater.cc:503] Master 127.0.97.62:34195 was elected leader, sending a full tablet report...
I20260926 03:04:27.184368 20706 ts_tablet_manager.cc:1428] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260926 03:04:27.184734 20708 raft_consensus.cc:348] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.185137 20708 raft_consensus.cc:374] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:27.185266 20708 raft_consensus.cc:729] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2693836d49a4d61b967ea0762f04594, State: Initialized, Role: FOLLOWER
I20260926 03:04:27.185871 20708 consensus_queue.cc:260] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.186942 20558 heartbeater.cc:503] Master 127.0.97.62:34195 was elected leader, sending a full tablet report...
I20260926 03:04:27.187564 20708 ts_tablet_manager.cc:1428] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260926 03:04:27.262449 20713 raft_consensus.cc:482] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:04:27.262795 20713 raft_consensus.cc:504] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.264761 20713 leader_election.cc:288] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5025bbd064f54047a1a282493c2651fd (127.0.97.1:39767), e2693836d49a4d61b967ea0762f04594 (127.0.97.2:41951)
I20260926 03:04:27.271329 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.265073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.97.3:59839)
0926 03:04:27.265258 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.265277 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:27.265415 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:27.265962 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:27.265966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:27.265992 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:27.266143 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.266151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.267490 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.267496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.269329 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.269352 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.270580 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.270586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.270783 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.270815 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 03:04:27.270914 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:04:27.271057 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":33}
I20260926 03:04:27.271358 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.265252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59839 (local address 127.0.97.1:39767)
0926 03:04:27.265439 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.265445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.265471 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:04:27.265562 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.265568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.265662 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.265760 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:27.266350 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.267367 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.269492 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.270429 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.270522 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.270597 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.270957 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.271079 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:04:27.271715 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.265688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41029 (local address 127.0.97.2:41951)
0926 03:04:27.265873 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.265881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.266205 (+ 324us) server_negotiation.cc:407] Connection header received
0926 03:04:27.266338 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.266342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.266412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.266479 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:04:27.267037 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.267911 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.269934 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.270793 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.270926 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.271006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.271341 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.271470 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:04:27.272034 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.265433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41951 (local address 127.0.97.3:41029)
0926 03:04:27.266104 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.266117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:04:27.266211 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:04:27.266524 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:04:27.266528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:04:27.266551 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:04:27.266898 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.266909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.268042 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.268046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.269760 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:04:27.269775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.270982 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.270987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:04:27.271159 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.271192 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 03:04:27.271283 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:04:27.271577 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":26,"thread_start_us":369,"threads_started":1}
I20260926 03:04:27.272465 20382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0434132134124b4a904205653d400" candidate_uuid: "89627965962e464e864b1d4144001d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5025bbd064f54047a1a282493c2651fd" is_pre_election: true
I20260926 03:04:27.273077 20382 raft_consensus.cc:2454] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89627965962e464e864b1d4144001d29 in term 0.
I20260926 03:04:27.273114 20513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0434132134124b4a904205653d400" candidate_uuid: "89627965962e464e864b1d4144001d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2693836d49a4d61b967ea0762f04594" is_pre_election: true
I20260926 03:04:27.273626 20513 raft_consensus.cc:2454] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89627965962e464e864b1d4144001d29 in term 0.
I20260926 03:04:27.273677 20579 leader_election.cc:302] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5025bbd064f54047a1a282493c2651fd, 89627965962e464e864b1d4144001d29; no voters:
I20260926 03:04:27.274042 20713 raft_consensus.cc:2781] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:04:27.274190 20713 raft_consensus.cc:482] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:04:27.274297 20713 raft_consensus.cc:3037] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:27.276387 20713 raft_consensus.cc:504] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.276954 20713 leader_election.cc:288] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [CANDIDATE]: Term 1 election: Requested vote from peers 5025bbd064f54047a1a282493c2651fd (127.0.97.1:39767), e2693836d49a4d61b967ea0762f04594 (127.0.97.2:41951)
I20260926 03:04:27.277391 20382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0434132134124b4a904205653d400" candidate_uuid: "89627965962e464e864b1d4144001d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5025bbd064f54047a1a282493c2651fd"
I20260926 03:04:27.277416 20513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b0434132134124b4a904205653d400" candidate_uuid: "89627965962e464e864b1d4144001d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2693836d49a4d61b967ea0762f04594"
I20260926 03:04:27.277580 20382 raft_consensus.cc:3037] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:27.277606 20513 raft_consensus.cc:3037] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:27.279929 20513 raft_consensus.cc:2454] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89627965962e464e864b1d4144001d29 in term 1.
I20260926 03:04:27.280066 20382 raft_consensus.cc:2454] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89627965962e464e864b1d4144001d29 in term 1.
I20260926 03:04:27.280323 20579 leader_election.cc:302] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89627965962e464e864b1d4144001d29, e2693836d49a4d61b967ea0762f04594; no voters:
I20260926 03:04:27.280647 20713 raft_consensus.cc:2781] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:27.281138 20713 raft_consensus.cc:686] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 1 LEADER]: Becoming Leader. State: Replica: 89627965962e464e864b1d4144001d29, State: Running, Role: LEADER
I20260926 03:04:27.281656 20713 consensus_queue.cc:237] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } }
I20260926 03:04:27.285997 20241 catalog_manager.cc:5495] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89627965962e464e864b1d4144001d29 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "89627965962e464e864b1d4144001d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89627965962e464e864b1d4144001d29" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 } health_report { overall_health: UNKNOWN } } }
W20260926 03:04:27.299618 20689 tablet.cc:2369] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:04:27.312297 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.307005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.0.97.3:45043)
0926 03:04:27.307208 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.307214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.307301 (+ 87us) server_negotiation.cc:407] Connection header received
0926 03:04:27.307556 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.307560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.307637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.307754 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:04:27.308200 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.309136 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.310508 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.310894 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.310985 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.311406 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:04:27.311879 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:04:27.311983 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.312058 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.312116 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:04:27.320168 20513 raft_consensus.cc:1261] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 1 FOLLOWER]: Refusing update from remote peer 89627965962e464e864b1d4144001d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:27.320408 20382 raft_consensus.cc:1261] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 1 FOLLOWER]: Refusing update from remote peer 89627965962e464e864b1d4144001d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:04:27.321192 20716 consensus_queue.cc:1035] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2693836d49a4d61b967ea0762f04594" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:27.321565 20713 consensus_queue.cc:1035] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5025bbd064f54047a1a282493c2651fd" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:04:27.336285 20719 mvcc.cc:205] Tried to move back new op lower bound from 7333445088533729280 to 7333445088389390336. Current Snapshot: MvccSnapshot[applied={T|T < 7333445088533729280}]
W20260926 03:04:27.344650 20559 tablet.cc:2369] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:04:27.359170 20241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:27.359465 20241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:04:27.363130 20241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e942a1eda464bcda857602f3a85c4b7: Sending DeleteTablet for 3 replicas of tablet a4b0434132134124b4a904205653d400
I20260926 03:04:27.364437 20623 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b0434132134124b4a904205653d400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:27 UTC) from {username='slave'} at 127.0.0.1:39464
I20260926 03:04:27.365128 20362 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b0434132134124b4a904205653d400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:27 UTC) from {username='slave'} at 127.0.0.1:37322
I20260926 03:04:27.365289 20493 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b0434132134124b4a904205653d400 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:04:27 UTC) from {username='slave'} at 127.0.0.1:57330
I20260926 03:04:27.365953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20300
I20260926 03:04:27.375366 20731 tablet_replica.cc:330] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: stopping tablet replica
I20260926 03:04:27.375681 20729 tablet_replica.cc:330] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: stopping tablet replica
I20260926 03:04:27.375931 20731 raft_consensus.cc:2229] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:27.376267 20729 raft_consensus.cc:2229] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:04:27.376544 20731 raft_consensus.cc:2258] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:27.377084 20729 raft_consensus.cc:2258] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:27.379439 20731 ts_tablet_manager.cc:1905] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:27.379441 20729 ts_tablet_manager.cc:1905] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:27.386934 20729 ts_tablet_manager.cc:1918] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:27.387193 20729 log.cc:1196] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-2/wal/wals/a4b0434132134124b4a904205653d400
I20260926 03:04:27.387948 20729 ts_tablet_manager.cc:1939] T a4b0434132134124b4a904205653d400 P 89627965962e464e864b1d4144001d29: Deleting consensus metadata
I20260926 03:04:27.387915 20731 ts_tablet_manager.cc:1918] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:27.388083 20731 log.cc:1196] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-1/wal/wals/a4b0434132134124b4a904205653d400
I20260926 03:04:27.388671 20731 ts_tablet_manager.cc:1939] T a4b0434132134124b4a904205653d400 P e2693836d49a4d61b967ea0762f04594: Deleting consensus metadata
I20260926 03:04:27.390105 20227 catalog_manager.cc:4841] TS 89627965962e464e864b1d4144001d29 (127.0.97.3:45043): tablet a4b0434132134124b4a904205653d400 (table test-table [id=a8357c6e0b6a4cec967ab9356616b098]) successfully deleted
I20260926 03:04:27.390925 20228 catalog_manager.cc:4841] TS e2693836d49a4d61b967ea0762f04594 (127.0.97.2:41951): tablet a4b0434132134124b4a904205653d400 (table test-table [id=a8357c6e0b6a4cec967ab9356616b098]) successfully deleted
I20260926 03:04:27.391343 20730 tablet_replica.cc:330] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: stopping tablet replica
I20260926 03:04:27.391920 20730 raft_consensus.cc:2229] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:04:27.392467 20730 raft_consensus.cc:2258] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:04:27.394703 20730 ts_tablet_manager.cc:1905] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:04:27.399168 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.368205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.0.97.1:39767)
0926 03:04:27.368445 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.368452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.391009 (+ 22557us) server_negotiation.cc:407] Connection header received
0926 03:04:27.391456 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.391463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.391601 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.391720 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:27.392219 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.393510 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.395360 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.396033 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.396175 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.397450 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:27.397461 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:27.397479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:27.398252 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:27.398312 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:27.398346 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:27.398503 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.398670 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.398783 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:04:27.400120 20422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:27.502722 20730 ts_tablet_manager.cc:1918] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:04:27.503003 20730 log.cc:1196] T a4b0434132134124b4a904205653d400 P 5025bbd064f54047a1a282493c2651fd: Deleting WAL directory at /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790391557890803-388-0/minicluster-data/ts-0/wal/wals/a4b0434132134124b4a904205653d400
I20260926 03:04:27.503392 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20300
I20260926 03:04:27.522092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20431
W20260926 03:04:27.522187 20228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39767 (error 108)
I20260926 03:04:27.524423 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.523835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37344)
0926 03:04:27.524099 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.524232 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 03:04:27.524698 20228 catalog_manager.cc:4584] TS 5025bbd064f54047a1a282493c2651fd (127.0.97.1:39767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4b0434132134124b4a904205653d400: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
I20260926 03:04:27.534102 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.522932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.0.97.2:41951)
0926 03:04:27.523144 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.523152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.523231 (+ 79us) server_negotiation.cc:407] Connection header received
0926 03:04:27.523585 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.523593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.523697 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.523810 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:27.524314 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.525409 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.526698 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.527104 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.527254 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.532584 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:27.532594 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:27.532609 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:27.533310 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:27.533370 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:27.533391 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:27.533512 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.533668 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.533777 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:04:27.534806 20553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:27.578686 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.578035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37360)
0926 03:04:27.578384 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.578490 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260926 03:04:27.579824 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.579403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37374)
0926 03:04:27.579592 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.579668 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 03:04:27.637506 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20431
I20260926 03:04:27.640489 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.639977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37386)
0926 03:04:27.640194 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.640336 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:04:27.641731 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.641291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37390)
0926 03:04:27.641466 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.641572 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 03:04:27.655757 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20561
I20260926 03:04:27.663619 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.656633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.0.97.3:45043)
0926 03:04:27.656843 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:27.656849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:27.656900 (+ 51us) server_negotiation.cc:407] Connection header received
0926 03:04:27.656986 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:27.657022 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:27.657133 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:27.657253 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:27.657613 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.658860 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.660052 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:27.660744 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:27.660884 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:27.661981 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:27.661992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:27.662009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:27.662827 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:27.662891 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:27.662916 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:27.663040 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:27.663114 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 03:04:27.663211 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 03:04:27.664403 20683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:27.722419 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.721869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37396)
0926 03:04:27.722097 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.722220 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:04:27.723552 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.723107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37402)
0926 03:04:27.723320 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.723407 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 03:04:27.785065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20561
I20260926 03:04:27.803741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20210
I20260926 03:04:27.804832 20271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:27.912981 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:27.912384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39767 (local address 127.0.0.1:37414)
0926 03:04:27.912595 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 03:04:27.912700 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:04:27.913138 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20210
2026-09-26T03:04:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2833 ms)
[----------] 6 tests from EncryptionPolicyTest (21835 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:28 dist-test-slave-fccw krb5kdc[20740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:28 dist-test-slave-fccw krb5kdc[20740](info): set up 2 sockets
Sep 26 03:04:28 dist-test-slave-fccw krb5kdc[20740](info): commencing operation
krb5kdc: starting...
W20260926 03:04:29.025449 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:29 dist-test-slave-fccw krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:29.268524 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:29.595566 20753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:29.595937 20753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:29.596109 20753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:29.603292 20753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:29.603441 20753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:29.603534 20753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:29.603641 20753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:29.603720 20753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:29.613472 20753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45563
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:29.614768 20753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:29.616060 20753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:29.627615 20761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:29.627692 20758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:29.627673 20759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:29.628561 20753 server_base.cc:1034] running on GCE node
I20260926 03:04:29.629345 20753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:29.630781 20753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:29.631982 20753 hybrid_clock.cc:648] HybridClock initialized: now 1790391869631929 us; error 49 us; skew 500 ppm
I20260926 03:04:29.632326 20753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:29 dist-test-slave-fccw krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:29.635383 20753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:29.636722 20753 webserver.cc:460] Webserver started at http://127.0.0.1:44741/ using document root <none> and password file <none>
I20260926 03:04:29.637466 20753 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:29.637590 20753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:29.637912 20753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:29.640841 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "c3cf666872344114b04fb4154f6de26c"
format_stamp: "Formatted at 2026-09-26 03:04:29 on dist-test-slave-fccw"
server_key: "f7ce5dd2feb93a28bb4ba908fd5d5251"
server_key_iv: "fe38d751dc720fbd204d7a128ce21716"
server_key_version: "encryptionkey@0"
I20260926 03:04:29.641667 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "c3cf666872344114b04fb4154f6de26c"
format_stamp: "Formatted at 2026-09-26 03:04:29 on dist-test-slave-fccw"
server_key: "f7ce5dd2feb93a28bb4ba908fd5d5251"
server_key_iv: "fe38d751dc720fbd204d7a128ce21716"
server_key_version: "encryptionkey@0"
I20260926 03:04:29.646775 20753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 03:04:29.650152 20768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:29.651435 20753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:29.651602 20753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "c3cf666872344114b04fb4154f6de26c"
format_stamp: "Formatted at 2026-09-26 03:04:29 on dist-test-slave-fccw"
server_key: "f7ce5dd2feb93a28bb4ba908fd5d5251"
server_key_iv: "fe38d751dc720fbd204d7a128ce21716"
server_key_version: "encryptionkey@0"
I20260926 03:04:29.651787 20753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:29.667436 20753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:29.671214 20753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:29.671502 20753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:29.688889 20753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45563
I20260926 03:04:29.688915 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45563 every 8 connection(s)
I20260926 03:04:29.690354 20753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:29.691430 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20753
I20260926 03:04:29.691663 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:29.692309 388 external_mini_cluster.cc:1428] Setting key dde477f8d493100291618322d777787b
I20260926 03:04:29.694546 20820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:04:29 dist-test-slave-fccw krb5kdc[20740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:04:29.707700 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c: Bootstrap starting.
I20260926 03:04:29.711011 20820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:29.712000 20820 log.cc:824] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:29.714802 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:29.694385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43403 (local address 127.0.0.1:45563)
0926 03:04:29.695174 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:29.695185 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:29.695224 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:04:29.696845 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:29.696865 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:29.697314 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:29.697642 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:29.699429 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:29.700766 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:29.702498 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:29.703105 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:29.703231 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:29.707848 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:29.707859 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:29.707877 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:29.707929 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:29.710805 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:29.711642 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:29.711648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:29.711654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:29.711814 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:29.712274 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:29.712283 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:29.712285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:29.712755 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:29.713146 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:29.713602 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 03:04:29.713897 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":388,"threads_started":1}
I20260926 03:04:29.715016 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c: No bootstrap required, opened a new log
I20260926 03:04:29.718757 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:29.722775 20820 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf666872344114b04fb4154f6de26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45563 } }
I20260926 03:04:29.723136 20820 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:29.723237 20820 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cf666872344114b04fb4154f6de26c, State: Initialized, Role: FOLLOWER
I20260926 03:04:29.723793 20820 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf666872344114b04fb4154f6de26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45563 } }
I20260926 03:04:29.724017 20820 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:29.724148 20820 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:29.724349 20820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:29.726667 20820 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf666872344114b04fb4154f6de26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45563 } }
I20260926 03:04:29.727118 20820 leader_election.cc:302] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3cf666872344114b04fb4154f6de26c; no voters:
I20260926 03:04:29.727836 20820 leader_election.cc:288] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:29.727952 20829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:29.728955 20829 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [term 1 LEADER]: Becoming Leader. State: Replica: c3cf666872344114b04fb4154f6de26c, State: Running, Role: LEADER
I20260926 03:04:29.729732 20829 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf666872344114b04fb4154f6de26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45563 } }
I20260926 03:04:29.731146 20820 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:29.732410 20832 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3cf666872344114b04fb4154f6de26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf666872344114b04fb4154f6de26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45563 } } }
I20260926 03:04:29.732815 20832 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:29.733882 20831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3cf666872344114b04fb4154f6de26c. Latest consensus state: current_term: 1 leader_uuid: "c3cf666872344114b04fb4154f6de26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cf666872344114b04fb4154f6de26c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45563 } } }
I20260926 03:04:29.734216 20831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:29.738145 20835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:29.744148 20835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:29.745839 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:29.727775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37383 (local address 127.0.0.1:45563)
0926 03:04:29.728004 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:29.728013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:29.728550 (+ 537us) server_negotiation.cc:407] Connection header received
0926 03:04:29.728733 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:29.728740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:29.728831 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:29.728950 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:29.730757 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:29.731958 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:29.733681 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:29.734458 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:29.734609 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:29.737481 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:29.737486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:29.737489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:29.737518 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:29.741484 (+ 3966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:29.742521 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:29.742527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:29.742530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:29.742839 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:29.743456 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:29.743461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:29.743468 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:29.743820 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:29.744169 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:29.745333 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0926 03:04:29.745504 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 03:04:29.753592 20835 catalog_manager.cc:1347] Generated new cluster ID: b781b2f1f7714601a14206e29fc9c5b6
I20260926 03:04:29.753701 20835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:29.764839 20835 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:29.766047 20835 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:29.773607 20835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3cf666872344114b04fb4154f6de26c: Generated new TSK 0
I20260926 03:04:29.774403 20835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:29.787892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20753
I20260926 03:04:29.789731 20815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:29.877985 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20753
W20260926 03:04:29.892103 20738 connection.cc:535] client connection to 127.0.0.1:45563 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45563): Connection reset by peer (error 104)
2026-09-26T03:04:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:29 dist-test-slave-fccw krb5kdc[20852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:29 dist-test-slave-fccw krb5kdc[20852](info): set up 2 sockets
Sep 26 03:04:29 dist-test-slave-fccw krb5kdc[20852](info): commencing operation
krb5kdc: starting...
W20260926 03:04:30.973331 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:31 dist-test-slave-fccw krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:31.218034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:31.541741 20865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:31.542079 20865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:31.542253 20865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:31.549321 20865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:31.549484 20865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:31.549597 20865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:31.549705 20865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:31.549777 20865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:31.559448 20865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:31.560775 20865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:31.562122 20865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:31.574060 20870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:31.574106 20871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:31.574132 20873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:31.575124 20865 server_base.cc:1034] running on GCE node
I20260926 03:04:31.575809 20865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:31.577170 20865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:31.578382 20865 hybrid_clock.cc:648] HybridClock initialized: now 1790391871578340 us; error 29 us; skew 500 ppm
I20260926 03:04:31.578753 20865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:31 dist-test-slave-fccw krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:31.581739 20865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:31.583236 20865 webserver.cc:460] Webserver started at http://127.0.0.1:33189/ using document root <none> and password file <none>
I20260926 03:04:31.583915 20865 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:31.584059 20865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:31.584367 20865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:31.587366 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "5f766e8ad42f48149a5929c91e360353"
format_stamp: "Formatted at 2026-09-26 03:04:31 on dist-test-slave-fccw"
server_key: "0410db86eed672857a012e1d49f92a86"
server_key_iv: "777ee89e774904d50512b1b2345df19c"
server_key_version: "encryptionkey@0"
I20260926 03:04:31.588125 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "5f766e8ad42f48149a5929c91e360353"
format_stamp: "Formatted at 2026-09-26 03:04:31 on dist-test-slave-fccw"
server_key: "0410db86eed672857a012e1d49f92a86"
server_key_iv: "777ee89e774904d50512b1b2345df19c"
server_key_version: "encryptionkey@0"
I20260926 03:04:31.593448 20865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 03:04:31.596925 20880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:31.598444 20865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:31.598663 20865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "5f766e8ad42f48149a5929c91e360353"
format_stamp: "Formatted at 2026-09-26 03:04:31 on dist-test-slave-fccw"
server_key: "0410db86eed672857a012e1d49f92a86"
server_key_iv: "777ee89e774904d50512b1b2345df19c"
server_key_version: "encryptionkey@0"
I20260926 03:04:31.598876 20865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:31.621939 20865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:31.626135 20865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:31.626431 20865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:31.644927 20865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35453
I20260926 03:04:31.644961 20931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35453 every 8 connection(s)
I20260926 03:04:31.646317 20865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:31.650446 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:31.650625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20865
I20260926 03:04:31.650811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:31.651321 388 external_mini_cluster.cc:1428] Setting key 2e3af1acc4fc58af502b043763d300ac
I20260926 03:04:31.663758 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353: Bootstrap starting.
I20260926 03:04:31.665378 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:31.653349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33239 (local address 127.0.0.1:35453)
0926 03:04:31.654177 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:31.654187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:31.654226 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:04:31.656476 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:31.656498 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:31.656905 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:31.657319 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:31.657858 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:31.659096 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:31.661100 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:31.661805 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:31.661954 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:31.663175 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:31.663182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:31.663195 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:31.663708 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:31.663794 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:31.663822 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:31.663988 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:31.664250 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 03:04:31.664401 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":392,"threads_started":1}
I20260926 03:04:31.666963 20932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:31.668165 20932 log.cc:824] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:31.669049 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:31.671293 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353: No bootstrap required, opened a new log
I20260926 03:04:31.680635 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:31.674120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52645 (local address 127.0.0.1:35453)
0926 03:04:31.674308 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:31.674313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:31.674650 (+ 337us) server_negotiation.cc:407] Connection header received
0926 03:04:31.674783 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:31.674793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:31.674875 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:31.674985 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:31.675487 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:31.676590 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:31.678021 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:31.678717 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:31.678866 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:31.679875 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:31.679882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:31.679886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:31.679907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:31.679969 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:31.679994 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:31.680110 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:31.680246 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 03:04:31.680301 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 03:04:31.681391 20932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f766e8ad42f48149a5929c91e360353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260926 03:04:31.681771 20932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:31.681895 20932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f766e8ad42f48149a5929c91e360353, State: Initialized, Role: FOLLOWER
I20260926 03:04:31.682552 20932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f766e8ad42f48149a5929c91e360353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260926 03:04:31.682789 20932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:31.682943 20932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:31.683141 20932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:31.685408 20932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f766e8ad42f48149a5929c91e360353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260926 03:04:31.685949 20932 leader_election.cc:302] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f766e8ad42f48149a5929c91e360353; no voters:
I20260926 03:04:31.686519 20932 leader_election.cc:288] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:31.686751 20942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:31.688248 20942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [term 1 LEADER]: Becoming Leader. State: Replica: 5f766e8ad42f48149a5929c91e360353, State: Running, Role: LEADER
I20260926 03:04:31.688702 20932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:31.688935 20942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f766e8ad42f48149a5929c91e360353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260926 03:04:31.691728 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f766e8ad42f48149a5929c91e360353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f766e8ad42f48149a5929c91e360353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } } }
I20260926 03:04:31.692030 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f766e8ad42f48149a5929c91e360353. Latest consensus state: current_term: 1 leader_uuid: "5f766e8ad42f48149a5929c91e360353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f766e8ad42f48149a5929c91e360353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } } }
I20260926 03:04:31.692245 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:31.692346 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:31.693440 20952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:31.700537 20952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:31.709803 20952 catalog_manager.cc:1347] Generated new cluster ID: e42853e8881248f597cb0cf35564ff2c
I20260926 03:04:31.709935 20952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:31.721329 20952 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:31.722601 20952 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:31.731679 20952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f766e8ad42f48149a5929c91e360353: Generated new TSK 0
I20260926 03:04:31.732497 20952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:31.742177 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20865
W20260926 03:04:31.743273 20927 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33239
W20260926 03:04:31.744061 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:31.744246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20865
W20260926 03:04:31.761070 20850 connection.cc:535] client connection to 127.0.0.1:35453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35453): Connection reset by peer (error 104)
2026-09-26T03:04:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:31 dist-test-slave-fccw krb5kdc[20963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:31 dist-test-slave-fccw krb5kdc[20963](info): set up 2 sockets
Sep 26 03:04:31 dist-test-slave-fccw krb5kdc[20963](info): commencing operation
krb5kdc: starting...
W20260926 03:04:32.837334 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:32 dist-test-slave-fccw krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:33.087715 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:33.419180 20976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:33.419457 20976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:33.419545 20976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:33.426860 20976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:33.426978 20976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:33.427022 20976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:33.427074 20976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:33.427109 20976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:33.436720 20976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:33.438046 20976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:33.439335 20976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:33.450976 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:33.451027 20981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:33.451053 20984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:33.452059 20976 server_base.cc:1034] running on GCE node
I20260926 03:04:33.452720 20976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:33.454216 20976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:33.455423 20976 hybrid_clock.cc:648] HybridClock initialized: now 1790391873455382 us; error 37 us; skew 500 ppm
I20260926 03:04:33.455763 20976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:33 dist-test-slave-fccw krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:33.463181 20976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:33.464676 20976 webserver.cc:460] Webserver started at http://127.0.0.1:34529/ using document root <none> and password file <none>
I20260926 03:04:33.465422 20976 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:33.465536 20976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:33.465807 20976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:33.468863 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "be7f0e4284a54521986d9897b367ba3a"
format_stamp: "Formatted at 2026-09-26 03:04:33 on dist-test-slave-fccw"
server_key: "086ea0b31b26e8243adbace4eda601c9"
server_key_iv: "c38397444c1581065cd67746f0099553"
server_key_version: "encryptionkey@0"
I20260926 03:04:33.469688 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "be7f0e4284a54521986d9897b367ba3a"
format_stamp: "Formatted at 2026-09-26 03:04:33 on dist-test-slave-fccw"
server_key: "086ea0b31b26e8243adbace4eda601c9"
server_key_iv: "c38397444c1581065cd67746f0099553"
server_key_version: "encryptionkey@0"
I20260926 03:04:33.474918 20976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 03:04:33.478423 20991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:33.479712 20976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:04:33.479895 20976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "be7f0e4284a54521986d9897b367ba3a"
format_stamp: "Formatted at 2026-09-26 03:04:33 on dist-test-slave-fccw"
server_key: "086ea0b31b26e8243adbace4eda601c9"
server_key_iv: "c38397444c1581065cd67746f0099553"
server_key_version: "encryptionkey@0"
I20260926 03:04:33.480064 20976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:33.485772 20976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:33.490100 20976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:33.490412 20976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:33.508765 20976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36555
I20260926 03:04:33.508800 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36555 every 8 connection(s)
I20260926 03:04:33.510206 20976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:33.512683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 20976
I20260926 03:04:33.512871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:33.513465 388 external_mini_cluster.cc:1428] Setting key 22448a99310cc20e10f186cec78c2be3
I20260926 03:04:33.514526 21043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:33.521251 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:33.515236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41069 (local address 127.0.0.1:36555)
0926 03:04:33.516024 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:33.516036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:33.516085 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:04:33.517740 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:33.517762 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:33.518106 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:33.518445 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:33.519514 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:33.519520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:33.519532 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:33.519546 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:33.519621 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:33.519640 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:33.519811 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:33.519911 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 03:04:33.520123 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":355,"threads_started":1}
I20260926 03:04:33.525251 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:33.528575 21043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a: Bootstrap starting.
I20260926 03:04:33.532903 21043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:33.532936 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:33.530645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60853 (local address 127.0.0.1:36555)
0926 03:04:33.530845 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:33.530851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:33.531074 (+ 223us) server_negotiation.cc:407] Connection header received
0926 03:04:33.531257 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:33.531261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:33.531307 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:33.531397 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:33.532315 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:33.532319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:33.532322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:33.532350 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:33.532385 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:33.532399 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:33.532481 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:33.532611 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 03:04:33.532652 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:04:33.534269 21043 log.cc:824] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:33.536847 21043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a: No bootstrap required, opened a new log
I20260926 03:04:33.543889 21043 raft_consensus.cc:348] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7f0e4284a54521986d9897b367ba3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36555 } }
I20260926 03:04:33.544296 21043 raft_consensus.cc:374] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:33.544462 21043 raft_consensus.cc:729] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7f0e4284a54521986d9897b367ba3a, State: Initialized, Role: FOLLOWER
I20260926 03:04:33.545082 21043 consensus_queue.cc:260] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7f0e4284a54521986d9897b367ba3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36555 } }
I20260926 03:04:33.545315 21043 raft_consensus.cc:388] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:33.545466 21043 raft_consensus.cc:482] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:33.545651 21043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:33.547763 21043 raft_consensus.cc:504] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7f0e4284a54521986d9897b367ba3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36555 } }
I20260926 03:04:33.548264 21043 leader_election.cc:302] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be7f0e4284a54521986d9897b367ba3a; no voters:
I20260926 03:04:33.548907 21043 leader_election.cc:288] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:33.549067 21053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:33.550424 21053 raft_consensus.cc:686] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [term 1 LEADER]: Becoming Leader. State: Replica: be7f0e4284a54521986d9897b367ba3a, State: Running, Role: LEADER
I20260926 03:04:33.550911 21053 consensus_queue.cc:237] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7f0e4284a54521986d9897b367ba3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36555 } }
I20260926 03:04:33.551318 21043 sys_catalog.cc:564] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:33.553812 21054 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be7f0e4284a54521986d9897b367ba3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7f0e4284a54521986d9897b367ba3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36555 } } }
I20260926 03:04:33.554211 21054 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:33.554678 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be7f0e4284a54521986d9897b367ba3a. Latest consensus state: current_term: 1 leader_uuid: "be7f0e4284a54521986d9897b367ba3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7f0e4284a54521986d9897b367ba3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36555 } } }
I20260926 03:04:33.554993 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:33.555176 21062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:33.561360 21062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:33.569859 21062 catalog_manager.cc:1347] Generated new cluster ID: 392f3b899e21430a9ddb0628d6f7c456
I20260926 03:04:33.569983 21062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:33.616496 21062 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:33.616983 21062 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:33.628274 21062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be7f0e4284a54521986d9897b367ba3a: Generated new TSK 0
I20260926 03:04:33.629369 21062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:33.644040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20976
W20260926 03:04:33.645169 21038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41069
W20260926 03:04:33.645937 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 20976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:33.646098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 20976
2026-09-26T03:04:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:33 dist-test-slave-fccw krb5kdc[21074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:33 dist-test-slave-fccw krb5kdc[21074](info): set up 2 sockets
Sep 26 03:04:33 dist-test-slave-fccw krb5kdc[21074](info): commencing operation
krb5kdc: starting...
W20260926 03:04:34.749482 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:34 dist-test-slave-fccw krb5kdc[21074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:34.988336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:35.318616 21087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:35.318984 21087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:35.319170 21087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:35.326382 21087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:35.326526 21087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:35.326628 21087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:35.326719 21087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:35.326783 21087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:35.336498 21087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:35.337896 21087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:35.339255 21087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:35.351105 21093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:35.351260 21095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:35.351635 21092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:35.352066 21087 server_base.cc:1034] running on GCE node
I20260926 03:04:35.352802 21087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:35.354220 21087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:35.355471 21087 hybrid_clock.cc:648] HybridClock initialized: now 1790391875355419 us; error 46 us; skew 500 ppm
I20260926 03:04:35.355798 21087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:35 dist-test-slave-fccw krb5kdc[21074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:35.359804 21087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:35.362946 21087 webserver.cc:460] Webserver started at http://127.0.0.1:45071/ using document root <none> and password file <none>
I20260926 03:04:35.363973 21087 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:35.364156 21087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:35.364563 21087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:35.369130 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "b30d06ee3ef1457db48bb992bc825d1b"
format_stamp: "Formatted at 2026-09-26 03:04:35 on dist-test-slave-fccw"
server_key: "f962b50254d5688e151772c102ded045"
server_key_iv: "7e17c8b4f5ad9b68d7b46b0d3b9025bf"
server_key_version: "encryptionkey@0"
I20260926 03:04:35.370208 21087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "b30d06ee3ef1457db48bb992bc825d1b"
format_stamp: "Formatted at 2026-09-26 03:04:35 on dist-test-slave-fccw"
server_key: "f962b50254d5688e151772c102ded045"
server_key_iv: "7e17c8b4f5ad9b68d7b46b0d3b9025bf"
server_key_version: "encryptionkey@0"
I20260926 03:04:35.377712 21087 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 03:04:35.383065 21102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:35.385356 21087 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260926 03:04:35.385613 21087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "b30d06ee3ef1457db48bb992bc825d1b"
format_stamp: "Formatted at 2026-09-26 03:04:35 on dist-test-slave-fccw"
server_key: "f962b50254d5688e151772c102ded045"
server_key_iv: "7e17c8b4f5ad9b68d7b46b0d3b9025bf"
server_key_version: "encryptionkey@0"
I20260926 03:04:35.385954 21087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:35.403429 21087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:35.407471 21087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:35.407706 21087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:35.426529 21087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33425
I20260926 03:04:35.426558 21153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33425 every 8 connection(s)
I20260926 03:04:35.428606 21087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:35.432861 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21087
I20260926 03:04:35.433075 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:35.433722 388 external_mini_cluster.cc:1428] Setting key d3489f287eff42a43f3d58eb28f4fa6f
I20260926 03:04:35.436179 21154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:04:35 dist-test-slave-fccw krb5kdc[21074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:04:35.451037 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b: Bootstrap starting.
I20260926 03:04:35.455147 21154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:35.456393 21154 log.cc:824] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:35.459689 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b: No bootstrap required, opened a new log
I20260926 03:04:35.460099 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:35.435525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59747 (local address 127.0.0.1:33425)
0926 03:04:35.436485 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:35.436509 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:35.436564 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:04:35.439408 (+ 2844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:35.439437 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:35.439993 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:35.440472 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:35.442481 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:35.444443 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:35.446278 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:35.447327 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:35.447506 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:35.451787 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:35.451797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:35.451815 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:35.451862 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:35.455890 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:35.456581 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:35.456590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:35.456597 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:35.456785 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:35.457248 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:35.457251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:35.457253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:35.457782 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:35.458069 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:35.458380 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 03:04:35.458779 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":385,"threads_started":1}
I20260926 03:04:35.463833 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:35.468112 21154 raft_consensus.cc:348] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30d06ee3ef1457db48bb992bc825d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33425 } }
I20260926 03:04:35.468581 21154 raft_consensus.cc:374] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:35.468698 21154 raft_consensus.cc:729] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30d06ee3ef1457db48bb992bc825d1b, State: Initialized, Role: FOLLOWER
I20260926 03:04:35.469439 21154 consensus_queue.cc:260] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30d06ee3ef1457db48bb992bc825d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33425 } }
I20260926 03:04:35.469740 21154 raft_consensus.cc:388] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:35.469894 21154 raft_consensus.cc:482] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:35.470081 21154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:35.473150 21154 raft_consensus.cc:504] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30d06ee3ef1457db48bb992bc825d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33425 } }
I20260926 03:04:35.473764 21154 leader_election.cc:302] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b30d06ee3ef1457db48bb992bc825d1b; no voters:
I20260926 03:04:35.474443 21154 leader_election.cc:288] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:35.474555 21164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:35.477072 21154 sys_catalog.cc:564] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:35.477126 21164 raft_consensus.cc:686] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [term 1 LEADER]: Becoming Leader. State: Replica: b30d06ee3ef1457db48bb992bc825d1b, State: Running, Role: LEADER
I20260926 03:04:35.477721 21164 consensus_queue.cc:237] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30d06ee3ef1457db48bb992bc825d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33425 } }
I20260926 03:04:35.482702 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b30d06ee3ef1457db48bb992bc825d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30d06ee3ef1457db48bb992bc825d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33425 } } }
I20260926 03:04:35.483844 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:35.482285 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b30d06ee3ef1457db48bb992bc825d1b. Latest consensus state: current_term: 1 leader_uuid: "b30d06ee3ef1457db48bb992bc825d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30d06ee3ef1457db48bb992bc825d1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33425 } } }
I20260926 03:04:35.484388 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:35.485589 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:35.468060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49591 (local address 127.0.0.1:33425)
0926 03:04:35.468272 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:35.468281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:35.468745 (+ 464us) server_negotiation.cc:407] Connection header received
0926 03:04:35.468920 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:35.468924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:35.469001 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:35.469144 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:35.471102 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:35.471886 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:35.473659 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:35.474141 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:35.474277 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:35.476860 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:35.476868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:35.476876 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:35.476925 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:35.480923 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:35.481912 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:35.481916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:35.481919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:35.482552 (+ 633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:35.483280 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:35.483283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:35.483285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:35.483736 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:35.485053 (+ 1317us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:35.485182 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 03:04:35.485343 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:04:35.488802 21173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:35.493405 21173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:35.501569 21173 catalog_manager.cc:1347] Generated new cluster ID: c6f4f30ae6c54754a0938a6d4c3256e6
I20260926 03:04:35.501674 21173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:35.519843 21173 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:35.521245 21173 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:35.528362 21173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b30d06ee3ef1457db48bb992bc825d1b: Generated new TSK 0
I20260926 03:04:35.529209 21173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:35.539420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21087
I20260926 03:04:35.540838 21149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:35.630182 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21087
2026-09-26T03:04:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:35 dist-test-slave-fccw krb5kdc[21186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:35 dist-test-slave-fccw krb5kdc[21186](info): set up 2 sockets
Sep 26 03:04:35 dist-test-slave-fccw krb5kdc[21186](info): commencing operation
krb5kdc: starting...
W20260926 03:04:36.733286 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:36 dist-test-slave-fccw krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:36.968412 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:37.289923 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:37.290182 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:37.290271 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:37.297854 21199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:37.297955 21199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:37.297998 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:37.298058 21199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:37.298098 21199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:37.307482 21199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:37.308609 21199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:37.309970 21199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:37.321341 21204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:37.321372 21205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:37.321309 21207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:37.322219 21199 server_base.cc:1034] running on GCE node
I20260926 03:04:37.322970 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:37.324272 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:37.325501 21199 hybrid_clock.cc:648] HybridClock initialized: now 1790391877325462 us; error 57 us; skew 500 ppm
I20260926 03:04:37.325831 21199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:37 dist-test-slave-fccw krb5kdc[21186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:37.328789 21199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:37.330132 21199 webserver.cc:460] Webserver started at http://127.0.0.1:44747/ using document root <none> and password file <none>
I20260926 03:04:37.330792 21199 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:37.330950 21199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:37.331243 21199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:37.334131 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "aee6d9c85db24cde9802bfdffc7dcca4"
format_stamp: "Formatted at 2026-09-26 03:04:37 on dist-test-slave-fccw"
server_key: "e15df2ec36f5d4f4e37908b6685d2fce"
server_key_iv: "d0f454c61d9b76123dda20fa7d6c7207"
server_key_version: "encryptionkey@0"
I20260926 03:04:37.334865 21199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "aee6d9c85db24cde9802bfdffc7dcca4"
format_stamp: "Formatted at 2026-09-26 03:04:37 on dist-test-slave-fccw"
server_key: "e15df2ec36f5d4f4e37908b6685d2fce"
server_key_iv: "d0f454c61d9b76123dda20fa7d6c7207"
server_key_version: "encryptionkey@0"
I20260926 03:04:37.339686 21199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 03:04:37.342873 21214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:37.344041 21199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:04:37.344214 21199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "aee6d9c85db24cde9802bfdffc7dcca4"
format_stamp: "Formatted at 2026-09-26 03:04:37 on dist-test-slave-fccw"
server_key: "e15df2ec36f5d4f4e37908b6685d2fce"
server_key_iv: "d0f454c61d9b76123dda20fa7d6c7207"
server_key_version: "encryptionkey@0"
I20260926 03:04:37.344403 21199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:37.363103 21199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:37.367072 21199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:37.367377 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:37.384929 21199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45431
I20260926 03:04:37.384948 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45431 every 8 connection(s)
I20260926 03:04:37.386274 21199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:37.389993 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:37.391171 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21199
I20260926 03:04:37.391392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:37.391927 388 external_mini_cluster.cc:1428] Setting key cb77d8c61cdffedec953229c427705e4
I20260926 03:04:37.403522 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4: Bootstrap starting.
I20260926 03:04:37.405148 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:37.393712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52133 (local address 127.0.0.1:45431)
0926 03:04:37.394516 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:37.394527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:37.394570 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:04:37.396222 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:37.396245 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:37.396633 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:37.396985 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:37.397561 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:37.398831 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:37.400772 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:37.401546 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:37.401682 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:37.402895 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:37.402903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:37.402916 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:37.403473 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:37.403548 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:37.403572 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:37.403705 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:37.403956 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 03:04:37.404197 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":374,"threads_started":1}
I20260926 03:04:37.406874 21266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:37.408150 21266 log.cc:824] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:37.408900 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:37.410759 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4: No bootstrap required, opened a new log
I20260926 03:04:37.420051 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:37.413311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35785 (local address 127.0.0.1:45431)
0926 03:04:37.413492 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:37.413498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:37.413910 (+ 412us) server_negotiation.cc:407] Connection header received
0926 03:04:37.414045 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:37.414049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:37.414128 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:37.414236 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:37.414717 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:37.415866 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:37.417264 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:37.417970 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:37.418104 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:37.419137 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:37.419144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:37.419148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:37.419168 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:37.419211 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:37.419234 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:37.419323 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:37.419459 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 03:04:37.419622 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:04:37.419600 21266 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260926 03:04:37.420217 21266 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:37.420324 21266 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee6d9c85db24cde9802bfdffc7dcca4, State: Initialized, Role: FOLLOWER
I20260926 03:04:37.420921 21266 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260926 03:04:37.421211 21266 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:37.421372 21266 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:37.421561 21266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:37.423570 21266 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260926 03:04:37.424052 21266 leader_election.cc:302] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee6d9c85db24cde9802bfdffc7dcca4; no voters:
I20260926 03:04:37.424576 21266 leader_election.cc:288] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:37.424702 21276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:37.425583 21276 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [term 1 LEADER]: Becoming Leader. State: Replica: aee6d9c85db24cde9802bfdffc7dcca4, State: Running, Role: LEADER
I20260926 03:04:37.426174 21276 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } }
I20260926 03:04:37.427132 21266 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:37.429121 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } } }
I20260926 03:04:37.429564 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:37.430371 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee6d9c85db24cde9802bfdffc7dcca4. Latest consensus state: current_term: 1 leader_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee6d9c85db24cde9802bfdffc7dcca4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45431 } } }
I20260926 03:04:37.430766 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:37.434504 21282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:37.440732 21282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:37.449306 21282 catalog_manager.cc:1347] Generated new cluster ID: de72db76273046e599e3c0a881a2e9f2
I20260926 03:04:37.449429 21282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:37.460916 21282 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:37.462176 21282 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:37.473222 21282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee6d9c85db24cde9802bfdffc7dcca4: Generated new TSK 0
I20260926 03:04:37.474045 21282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:37.482616 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21199
W20260926 03:04:37.483875 21261 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52133
W20260926 03:04:37.484563 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21199: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:37.484711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21199
2026-09-26T03:04:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:37 dist-test-slave-fccw krb5kdc[21297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:37 dist-test-slave-fccw krb5kdc[21297](info): set up 2 sockets
Sep 26 03:04:37 dist-test-slave-fccw krb5kdc[21297](info): commencing operation
krb5kdc: starting...
W20260926 03:04:38.593184 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:38 dist-test-slave-fccw krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:38.836834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:39.170387 21310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:39.170785 21310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:39.170953 21310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:39.178221 21310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:39.178381 21310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:39.178495 21310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:39.178606 21310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:39.178694 21310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:39.188208 21310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:39.189494 21310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:39.190748 21310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:39.202447 21315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:39.202543 21318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:39.202417 21316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:39.203615 21310 server_base.cc:1034] running on GCE node
I20260926 03:04:39.204416 21310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:39.205789 21310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:39.207017 21310 hybrid_clock.cc:648] HybridClock initialized: now 1790391879206986 us; error 49 us; skew 500 ppm
I20260926 03:04:39.207391 21310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:39 dist-test-slave-fccw krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:39.215977 21310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:39.217473 21310 webserver.cc:460] Webserver started at http://127.0.0.1:39905/ using document root <none> and password file <none>
I20260926 03:04:39.218151 21310 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:39.218297 21310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:39.218613 21310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:39.221678 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "e8d77d87b8fa4e5a864c057d25190937"
format_stamp: "Formatted at 2026-09-26 03:04:39 on dist-test-slave-fccw"
server_key: "02c736f04a9aea33bf42de72ce91d98e"
server_key_iv: "6f2b6590093500122a9b60054fd779ea"
server_key_version: "encryptionkey@0"
I20260926 03:04:39.222451 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "e8d77d87b8fa4e5a864c057d25190937"
format_stamp: "Formatted at 2026-09-26 03:04:39 on dist-test-slave-fccw"
server_key: "02c736f04a9aea33bf42de72ce91d98e"
server_key_iv: "6f2b6590093500122a9b60054fd779ea"
server_key_version: "encryptionkey@0"
I20260926 03:04:39.227399 21310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:04:39.230823 21325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:39.232071 21310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:04:39.232247 21310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "e8d77d87b8fa4e5a864c057d25190937"
format_stamp: "Formatted at 2026-09-26 03:04:39 on dist-test-slave-fccw"
server_key: "02c736f04a9aea33bf42de72ce91d98e"
server_key_iv: "6f2b6590093500122a9b60054fd779ea"
server_key_version: "encryptionkey@0"
I20260926 03:04:39.232448 21310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:39.238085 21310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:39.241812 21310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:39.242130 21310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:39.260000 21310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35073
I20260926 03:04:39.260018 21376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35073 every 8 connection(s)
I20260926 03:04:39.261425 21310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:39.264940 21377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:39.270260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21310
I20260926 03:04:39.270440 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:39.270849 388 external_mini_cluster.cc:1428] Setting key 28ed1cda60b0c0199568f458e4bbf3a4
I20260926 03:04:39.276117 21377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937: Bootstrap starting.
I20260926 03:04:39.279445 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:39.272581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36709 (local address 127.0.0.1:35073)
0926 03:04:39.273449 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:39.273463 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:39.273509 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:04:39.275777 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:39.275804 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:39.276226 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:39.276634 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:39.277584 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:39.277598 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:39.277615 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:39.277634 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:39.277719 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:39.277752 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:39.277986 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:39.278124 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 03:04:39.278315 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260926 03:04:39.279930 21377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:39.281188 21377 log.cc:824] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:39.283193 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:39.284088 21377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937: No bootstrap required, opened a new log
I20260926 03:04:39.289804 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:39.287524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34821 (local address 127.0.0.1:35073)
0926 03:04:39.287693 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:39.287699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:39.287951 (+ 252us) server_negotiation.cc:407] Connection header received
0926 03:04:39.288071 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:39.288076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:39.288137 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:39.288256 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:39.289128 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:39.289135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:39.289140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:39.289158 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:39.289207 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:39.289231 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:39.289367 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:39.289437 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 03:04:39.289495 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 03:04:39.292341 21377 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d77d87b8fa4e5a864c057d25190937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35073 } }
I20260926 03:04:39.292658 21377 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:39.292733 21377 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d77d87b8fa4e5a864c057d25190937, State: Initialized, Role: FOLLOWER
I20260926 03:04:39.293270 21377 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d77d87b8fa4e5a864c057d25190937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35073 } }
I20260926 03:04:39.293468 21377 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:39.293556 21377 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:39.293670 21377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:39.295694 21377 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d77d87b8fa4e5a864c057d25190937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35073 } }
I20260926 03:04:39.296159 21377 leader_election.cc:302] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d77d87b8fa4e5a864c057d25190937; no voters:
I20260926 03:04:39.296715 21377 leader_election.cc:288] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:39.296850 21387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:39.297564 21387 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [term 1 LEADER]: Becoming Leader. State: Replica: e8d77d87b8fa4e5a864c057d25190937, State: Running, Role: LEADER
I20260926 03:04:39.298130 21387 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d77d87b8fa4e5a864c057d25190937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35073 } }
I20260926 03:04:39.298857 21377 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:39.300802 21389 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d77d87b8fa4e5a864c057d25190937. Latest consensus state: current_term: 1 leader_uuid: "e8d77d87b8fa4e5a864c057d25190937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d77d87b8fa4e5a864c057d25190937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35073 } } }
I20260926 03:04:39.301307 21389 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:39.301242 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8d77d87b8fa4e5a864c057d25190937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d77d87b8fa4e5a864c057d25190937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35073 } } }
I20260926 03:04:39.301545 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:39.307173 21394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:39.311435 21394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:39.319665 21394 catalog_manager.cc:1347] Generated new cluster ID: 677813e9c5164388965d4fcb72c71e5b
I20260926 03:04:39.319780 21394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:39.360255 21394 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:39.360775 21394 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:39.367393 21394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8d77d87b8fa4e5a864c057d25190937: Generated new TSK 0
I20260926 03:04:39.368271 21394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:39.384285 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21310
W20260926 03:04:39.385362 21372 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36709
W20260926 03:04:39.386057 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21310: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:39.386193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21310
2026-09-26T03:04:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:39 dist-test-slave-fccw krb5kdc[21408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:39 dist-test-slave-fccw krb5kdc[21408](info): set up 2 sockets
Sep 26 03:04:39 dist-test-slave-fccw krb5kdc[21408](info): commencing operation
krb5kdc: starting...
W20260926 03:04:40.481385 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:40 dist-test-slave-fccw krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:40.720085 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:41.042198 21421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:41.042557 21421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:41.042717 21421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:41.050071 21421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:41.050230 21421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:41.050329 21421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:41.050446 21421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:41.050524 21421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:41.060439 21421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:41.061774 21421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:41.063117 21421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:41.074741 21426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:41.074817 21429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:41.074779 21427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:41.075547 21421 server_base.cc:1034] running on GCE node
I20260926 03:04:41.076339 21421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:41.077775 21421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:41.079022 21421 hybrid_clock.cc:648] HybridClock initialized: now 1790391881078987 us; error 46 us; skew 500 ppm
I20260926 03:04:41.079366 21421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:41 dist-test-slave-fccw krb5kdc[21408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:41.082413 21421 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:41.083925 21421 webserver.cc:460] Webserver started at http://127.0.0.1:40457/ using document root <none> and password file <none>
I20260926 03:04:41.084609 21421 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:41.084754 21421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:41.085139 21421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:41.088056 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "a0d8a8fb6f7f4215988e0247d47cab99"
format_stamp: "Formatted at 2026-09-26 03:04:41 on dist-test-slave-fccw"
server_key: "552d84be5a67dd34a64730f731bebb87"
server_key_iv: "a91f198f883f7211343a253ea0fd768e"
server_key_version: "encryptionkey@0"
I20260926 03:04:41.088824 21421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "a0d8a8fb6f7f4215988e0247d47cab99"
format_stamp: "Formatted at 2026-09-26 03:04:41 on dist-test-slave-fccw"
server_key: "552d84be5a67dd34a64730f731bebb87"
server_key_iv: "a91f198f883f7211343a253ea0fd768e"
server_key_version: "encryptionkey@0"
I20260926 03:04:41.094614 21421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 03:04:41.099443 21436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:41.101477 21421 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 03:04:41.101657 21421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "a0d8a8fb6f7f4215988e0247d47cab99"
format_stamp: "Formatted at 2026-09-26 03:04:41 on dist-test-slave-fccw"
server_key: "552d84be5a67dd34a64730f731bebb87"
server_key_iv: "a91f198f883f7211343a253ea0fd768e"
server_key_version: "encryptionkey@0"
I20260926 03:04:41.101953 21421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:41.115971 21421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:41.119886 21421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:41.120189 21421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:41.138818 21421 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32851
I20260926 03:04:41.138844 21487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32851 every 8 connection(s)
I20260926 03:04:41.140239 21421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:41.142483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21421
I20260926 03:04:41.142668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:41.143210 388 external_mini_cluster.cc:1428] Setting key 7f07ae94704df71e8c6d1add1b9491ad
I20260926 03:04:41.144640 21488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:04:41 dist-test-slave-fccw krb5kdc[21408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:04:41.158607 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99: Bootstrap starting.
I20260926 03:04:41.161871 21488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:41.162894 21488 log.cc:824] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:41.165880 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99: No bootstrap required, opened a new log
I20260926 03:04:41.166455 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:41.145030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:39395 (local address 127.0.0.1:32851)
0926 03:04:41.145872 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:41.145884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:41.145924 (+ 40us) server_negotiation.cc:407] Connection header received
0926 03:04:41.147500 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:41.147521 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:41.147908 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:41.148239 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:41.150086 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:41.151428 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:41.153229 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:41.153880 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:41.154010 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:41.158848 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:41.158856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:41.158871 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:41.158911 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:41.162107 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:41.163005 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:41.163017 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:41.163027 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:41.163196 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:41.163680 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:41.163683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:41.163685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:41.164222 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:41.164556 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:41.165104 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 03:04:41.165410 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":395,"threads_started":1}
I20260926 03:04:41.170598 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:41.174474 21488 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260926 03:04:41.175000 21488 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:41.175212 21488 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d8a8fb6f7f4215988e0247d47cab99, State: Initialized, Role: FOLLOWER
I20260926 03:04:41.175902 21488 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260926 03:04:41.176188 21488 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:41.176357 21488 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:41.176522 21488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:41.179353 21488 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260926 03:04:41.179858 21488 leader_election.cc:302] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0d8a8fb6f7f4215988e0247d47cab99; no voters:
I20260926 03:04:41.180491 21488 leader_election.cc:288] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:41.180619 21498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:41.182310 21498 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [term 1 LEADER]: Becoming Leader. State: Replica: a0d8a8fb6f7f4215988e0247d47cab99, State: Running, Role: LEADER
I20260926 03:04:41.183017 21488 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:41.183013 21498 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } }
I20260926 03:04:41.190948 21500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0d8a8fb6f7f4215988e0247d47cab99. Latest consensus state: current_term: 1 leader_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } } }
I20260926 03:04:41.191423 21500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:41.194129 21499 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0d8a8fb6f7f4215988e0247d47cab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32851 } } }
I20260926 03:04:41.195628 21499 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:41.195519 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:41.174998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:44909 (local address 127.0.0.1:32851)
0926 03:04:41.175169 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:41.175176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:41.175629 (+ 453us) server_negotiation.cc:407] Connection header received
0926 03:04:41.175809 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:41.175815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:41.175908 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:41.176020 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:41.177778 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:41.178984 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:41.180454 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:41.181312 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:41.181431 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:41.184648 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:41.184656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:41.184663 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:41.184709 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:41.189589 (+ 4880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:41.190697 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:41.190705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:41.190709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:41.190917 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:41.191554 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:41.191561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:41.191566 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:41.192042 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:41.192433 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:41.194754 (+ 2321us) server_negotiation.cc:299] Negotiation successful
0926 03:04:41.195006 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 03:04:41.198246 21507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:41.203907 21507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:41.210597 21516 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:04:41.212604 21507 catalog_manager.cc:1347] Generated new cluster ID: 773b273fc529404c9b7cb9c9d7727b79
I20260926 03:04:41.212713 21507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:41.220664 21507 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:41.221985 21507 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:41.232218 21507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0d8a8fb6f7f4215988e0247d47cab99: Generated new TSK 0
I20260926 03:04:41.233275 21507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:41.247314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21421
I20260926 03:04:41.248386 21483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:41.338322 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21421
W20260926 03:04:41.355674 21406 connection.cc:535] client connection to 127.0.0.1:32851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32851): Connection reset by peer (error 104)
2026-09-26T03:04:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:41 dist-test-slave-fccw krb5kdc[21521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:41 dist-test-slave-fccw krb5kdc[21521](info): set up 2 sockets
Sep 26 03:04:41 dist-test-slave-fccw krb5kdc[21521](info): commencing operation
krb5kdc: starting...
W20260926 03:04:42.425515 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:42 dist-test-slave-fccw krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:42.670197 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:43.008787 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:43.009131 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:43.009307 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:43.016667 21534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:43.016811 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:43.016893 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:43.017023 21534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:43.017091 21534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:43.026751 21534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:43.028045 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:43.029384 21534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:43.040910 21542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:43.040962 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:43.041181 21539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:43.041839 21534 server_base.cc:1034] running on GCE node
I20260926 03:04:43.042583 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:43.044003 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:43.045223 21534 hybrid_clock.cc:648] HybridClock initialized: now 1790391883045179 us; error 55 us; skew 500 ppm
I20260926 03:04:43.045550 21534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:43 dist-test-slave-fccw krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:43.048622 21534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:43.050092 21534 webserver.cc:460] Webserver started at http://127.0.0.1:33623/ using document root <none> and password file <none>
I20260926 03:04:43.050736 21534 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:43.050905 21534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:43.051231 21534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:43.054071 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "63e340e8f97c40d29e368ff824b6e9f0"
format_stamp: "Formatted at 2026-09-26 03:04:43 on dist-test-slave-fccw"
server_key: "98cdbea275fc638af0909df3b2897142"
server_key_iv: "4ff2a0147084fa7e0defb2f27bdd2527"
server_key_version: "encryptionkey@0"
I20260926 03:04:43.054800 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "63e340e8f97c40d29e368ff824b6e9f0"
format_stamp: "Formatted at 2026-09-26 03:04:43 on dist-test-slave-fccw"
server_key: "98cdbea275fc638af0909df3b2897142"
server_key_iv: "4ff2a0147084fa7e0defb2f27bdd2527"
server_key_version: "encryptionkey@0"
I20260926 03:04:43.059742 21534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 03:04:43.063043 21549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:43.064247 21534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:04:43.064458 21534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "63e340e8f97c40d29e368ff824b6e9f0"
format_stamp: "Formatted at 2026-09-26 03:04:43 on dist-test-slave-fccw"
server_key: "98cdbea275fc638af0909df3b2897142"
server_key_iv: "4ff2a0147084fa7e0defb2f27bdd2527"
server_key_version: "encryptionkey@0"
I20260926 03:04:43.064664 21534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:43.081306 21534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:43.085534 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:43.085824 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:43.108611 21534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44211
I20260926 03:04:43.108650 21600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44211 every 8 connection(s)
I20260926 03:04:43.110004 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:43.113196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21534
I20260926 03:04:43.113386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:43.113885 21601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:43.113989 388 external_mini_cluster.cc:1428] Setting key b2e794885fd649a0dabab7d998a35b68
I20260926 03:04:43.126324 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0: Bootstrap starting.
I20260926 03:04:43.127214 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:43.115731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:39219 (local address 127.0.0.1:44211)
0926 03:04:43.116528 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:43.116545 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:43.116864 (+ 319us) server_negotiation.cc:407] Connection header received
0926 03:04:43.118880 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:43.118899 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:43.119294 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:43.119684 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:43.120231 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:43.121589 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:43.122906 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:43.123562 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:43.123764 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:43.125075 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:43.125091 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:43.125105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:43.125598 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:43.125668 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:43.125692 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:43.125858 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:43.126131 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 03:04:43.126285 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":346,"threads_started":1}
I20260926 03:04:43.129578 21601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:43.130764 21601 log.cc:824] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:43.130959 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:43.133513 21601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0: No bootstrap required, opened a new log
I20260926 03:04:43.142184 21601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e340e8f97c40d29e368ff824b6e9f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260926 03:04:43.142575 21601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:43.142719 21601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e340e8f97c40d29e368ff824b6e9f0, State: Initialized, Role: FOLLOWER
I20260926 03:04:43.142992 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:43.135219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:55783 (local address 127.0.0.1:44211)
0926 03:04:43.135375 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:43.135378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:43.137503 (+ 2125us) server_negotiation.cc:407] Connection header received
0926 03:04:43.137683 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:43.137688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:43.137760 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:43.137877 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:43.138344 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:43.139115 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:43.140347 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:43.140788 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:43.140955 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:43.142095 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:43.142102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:43.142111 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:43.142127 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:43.142178 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:43.142194 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:43.142308 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:43.142501 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 03:04:43.142551 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:04:43.143421 21601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e340e8f97c40d29e368ff824b6e9f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260926 03:04:43.143734 21601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:43.143885 21601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:43.144068 21601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:43.146281 21601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e340e8f97c40d29e368ff824b6e9f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260926 03:04:43.146770 21601 leader_election.cc:302] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e340e8f97c40d29e368ff824b6e9f0; no voters:
I20260926 03:04:43.147341 21601 leader_election.cc:288] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:43.147439 21611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:43.148175 21611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 63e340e8f97c40d29e368ff824b6e9f0, State: Running, Role: LEADER
I20260926 03:04:43.148723 21611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e340e8f97c40d29e368ff824b6e9f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } }
I20260926 03:04:43.149400 21601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:43.151396 21613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e340e8f97c40d29e368ff824b6e9f0. Latest consensus state: current_term: 1 leader_uuid: "63e340e8f97c40d29e368ff824b6e9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e340e8f97c40d29e368ff824b6e9f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } } }
I20260926 03:04:43.151759 21613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:43.152386 21612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e340e8f97c40d29e368ff824b6e9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e340e8f97c40d29e368ff824b6e9f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44211 } } }
I20260926 03:04:43.152774 21612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:43.155525 21618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:43.161865 21618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:43.177275 21618 catalog_manager.cc:1347] Generated new cluster ID: e8ec3e2938ff45fda2c88aead095d4a8
I20260926 03:04:43.177425 21618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:43.194140 21618 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:43.196004 21618 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:43.205842 21618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e340e8f97c40d29e368ff824b6e9f0: Generated new TSK 0
I20260926 03:04:43.206776 21618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:43.220398 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21534
W20260926 03:04:43.221611 21596 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.114:39219
W20260926 03:04:43.222390 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21534: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:43.222541 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21534
W20260926 03:04:43.239296 21519 connection.cc:535] client connection to 127.0.0.1:44211 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44211): Connection reset by peer (error 104)
2026-09-26T03:04:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:43 dist-test-slave-fccw krb5kdc[21633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:43 dist-test-slave-fccw krb5kdc[21633](info): set up 2 sockets
Sep 26 03:04:43 dist-test-slave-fccw krb5kdc[21633](info): commencing operation
krb5kdc: starting...
W20260926 03:04:44.321237 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:44 dist-test-slave-fccw krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:44.573861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:44.895642 21646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:44.896013 21646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:44.896173 21646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:44.903141 21646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:44.903282 21646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:44.903365 21646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:44.903467 21646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:44.903533 21646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:44.912977 21646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:44.914234 21646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:44.915812 21646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:44.927244 21651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:44.927361 21652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:44.927924 21654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:44.929208 21646 server_base.cc:1034] running on GCE node
I20260926 03:04:44.929991 21646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:44.931318 21646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:44.932520 21646 hybrid_clock.cc:648] HybridClock initialized: now 1790391884932479 us; error 32 us; skew 500 ppm
I20260926 03:04:44.932858 21646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:44 dist-test-slave-fccw krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:44.940460 21646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:44.941939 21646 webserver.cc:460] Webserver started at http://127.0.0.1:34109/ using document root <none> and password file <none>
I20260926 03:04:44.942663 21646 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:44.942763 21646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:44.943061 21646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:44.945951 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "d5195b411eb64d579eb905684f6ddc36"
format_stamp: "Formatted at 2026-09-26 03:04:44 on dist-test-slave-fccw"
server_key: "8cecbaba12f1066f3eec6f1652284142"
server_key_iv: "956058213d5f6ee2e1c1fd54cb7a8305"
server_key_version: "encryptionkey@0"
I20260926 03:04:44.946745 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "d5195b411eb64d579eb905684f6ddc36"
format_stamp: "Formatted at 2026-09-26 03:04:44 on dist-test-slave-fccw"
server_key: "8cecbaba12f1066f3eec6f1652284142"
server_key_iv: "956058213d5f6ee2e1c1fd54cb7a8305"
server_key_version: "encryptionkey@0"
I20260926 03:04:44.951821 21646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 03:04:44.955143 21661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:44.956574 21646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:04:44.956760 21646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "d5195b411eb64d579eb905684f6ddc36"
format_stamp: "Formatted at 2026-09-26 03:04:44 on dist-test-slave-fccw"
server_key: "8cecbaba12f1066f3eec6f1652284142"
server_key_iv: "956058213d5f6ee2e1c1fd54cb7a8305"
server_key_version: "encryptionkey@0"
I20260926 03:04:44.956954 21646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:44.962710 21646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:44.966804 21646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:44.967108 21646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:44.985821 21646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33235
I20260926 03:04:44.985922 21712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33235 every 8 connection(s)
I20260926 03:04:44.987497 21646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:44.991153 21713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:44.997710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21646
I20260926 03:04:44.997910 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:44.998339 388 external_mini_cluster.cc:1428] Setting key a6c6909038db2c4514c6453c78026b68
I20260926 03:04:45.001693 21713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36: Bootstrap starting.
I20260926 03:04:45.005213 21713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:45.006228 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:45.000217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:56831 (local address 127.0.0.1:33235)
0926 03:04:45.001078 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:45.001090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:45.001132 (+ 42us) server_negotiation.cc:407] Connection header received
0926 03:04:45.002890 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:45.002935 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:45.003270 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:45.003588 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:45.004639 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:45.004645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:45.004658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:45.004678 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:45.004731 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:45.004748 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:45.004886 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:45.005115 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 03:04:45.005326 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":377,"threads_started":1}
I20260926 03:04:45.006316 21713 log.cc:824] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:45.009734 21713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36: No bootstrap required, opened a new log
I20260926 03:04:45.011567 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:45.017879 21713 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5195b411eb64d579eb905684f6ddc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260926 03:04:45.018358 21713 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:45.018355 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:45.015785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:48775 (local address 127.0.0.1:33235)
0926 03:04:45.015944 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:45.015948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:45.016382 (+ 434us) server_negotiation.cc:407] Connection header received
0926 03:04:45.016578 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:45.016584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:45.016643 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:45.016751 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:45.017703 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:45.017708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:45.017713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:45.017731 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:45.017778 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:45.017793 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:45.017941 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:45.017999 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 03:04:45.018059 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:04:45.018466 21713 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5195b411eb64d579eb905684f6ddc36, State: Initialized, Role: FOLLOWER
I20260926 03:04:45.019127 21713 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5195b411eb64d579eb905684f6ddc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260926 03:04:45.019373 21713 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:45.019546 21713 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:45.019733 21713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:45.021982 21713 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5195b411eb64d579eb905684f6ddc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260926 03:04:45.022476 21713 leader_election.cc:302] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5195b411eb64d579eb905684f6ddc36; no voters:
I20260926 03:04:45.023025 21713 leader_election.cc:288] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:45.023182 21723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:45.023847 21723 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [term 1 LEADER]: Becoming Leader. State: Replica: d5195b411eb64d579eb905684f6ddc36, State: Running, Role: LEADER
I20260926 03:04:45.024405 21723 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5195b411eb64d579eb905684f6ddc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260926 03:04:45.025285 21713 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:45.027397 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5195b411eb64d579eb905684f6ddc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5195b411eb64d579eb905684f6ddc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } } }
I20260926 03:04:45.028057 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:45.029028 21725 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5195b411eb64d579eb905684f6ddc36. Latest consensus state: current_term: 1 leader_uuid: "d5195b411eb64d579eb905684f6ddc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5195b411eb64d579eb905684f6ddc36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } } }
I20260926 03:04:45.029451 21725 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:45.029886 21732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:45.034966 21732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:45.043689 21732 catalog_manager.cc:1347] Generated new cluster ID: ea39dd382ad843ddbc1cd7220b36c54f
I20260926 03:04:45.043792 21732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:45.063995 21732 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:45.064477 21732 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:45.072140 21732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5195b411eb64d579eb905684f6ddc36: Generated new TSK 0
I20260926 03:04:45.073115 21732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:45.084370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21646
W20260926 03:04:45.085531 21708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.114:56831
W20260926 03:04:45.086258 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21646: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:45.086409 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21646
2026-09-26T03:04:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:45 dist-test-slave-fccw krb5kdc[21744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:45 dist-test-slave-fccw krb5kdc[21744](info): set up 2 sockets
Sep 26 03:04:45 dist-test-slave-fccw krb5kdc[21744](info): commencing operation
krb5kdc: starting...
W20260926 03:04:46.209431 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:46 dist-test-slave-fccw krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:46.453343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:46.776952 21757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:46.777288 21757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:46.777472 21757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:46.784677 21757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:46.784780 21757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:46.784855 21757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:46.784977 21757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:46.785123 21757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:46.795243 21757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:46.796572 21757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:46.797972 21757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:46.809700 21762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:46.809803 21765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:46.809803 21763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:46.810635 21757 server_base.cc:1034] running on GCE node
I20260926 03:04:46.811470 21757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:46.812912 21757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:46.814137 21757 hybrid_clock.cc:648] HybridClock initialized: now 1790391886814086 us; error 53 us; skew 500 ppm
I20260926 03:04:46.814486 21757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:46 dist-test-slave-fccw krb5kdc[21744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:46.817685 21757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:46.819159 21757 webserver.cc:460] Webserver started at http://127.0.0.1:40575/ using document root <none> and password file <none>
I20260926 03:04:46.819802 21757 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:46.819911 21757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:46.820168 21757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:46.823083 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "d8094236d1be4f7e842d568ecbcbf287"
format_stamp: "Formatted at 2026-09-26 03:04:46 on dist-test-slave-fccw"
server_key: "bdaa2b0e1f4d76cc347ee75f9bab7cdc"
server_key_iv: "77ce29c881f50412ecc5cfe22b49ce4b"
server_key_version: "encryptionkey@0"
I20260926 03:04:46.823758 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "d8094236d1be4f7e842d568ecbcbf287"
format_stamp: "Formatted at 2026-09-26 03:04:46 on dist-test-slave-fccw"
server_key: "bdaa2b0e1f4d76cc347ee75f9bab7cdc"
server_key_iv: "77ce29c881f50412ecc5cfe22b49ce4b"
server_key_version: "encryptionkey@0"
I20260926 03:04:46.828701 21757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:04:46.832095 21772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:46.833500 21757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 03:04:46.833681 21757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "d8094236d1be4f7e842d568ecbcbf287"
format_stamp: "Formatted at 2026-09-26 03:04:46 on dist-test-slave-fccw"
server_key: "bdaa2b0e1f4d76cc347ee75f9bab7cdc"
server_key_iv: "77ce29c881f50412ecc5cfe22b49ce4b"
server_key_version: "encryptionkey@0"
I20260926 03:04:46.833868 21757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:46.850869 21757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:46.855471 21757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:46.855767 21757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:46.874377 21757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40359
I20260926 03:04:46.874423 21823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40359 every 8 connection(s)
I20260926 03:04:46.875838 21757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:46.876202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21757
I20260926 03:04:46.876423 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:46.876879 388 external_mini_cluster.cc:1428] Setting key 9780012435675ce61e54cd75b18156f6
I20260926 03:04:46.880491 21824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 03:04:46 dist-test-slave-fccw krb5kdc[21744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 03:04:46.890866 21824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287: Bootstrap starting.
I20260926 03:04:46.894004 21824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:46.895010 21824 log.cc:824] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:46.897974 21824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287: No bootstrap required, opened a new log
I20260926 03:04:46.898813 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:46.878817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:60303 (local address 127.0.0.1:40359)
0926 03:04:46.879591 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:46.879604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:46.879643 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:04:46.881418 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:46.881440 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:46.881827 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:46.882194 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:46.883658 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:46.885051 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:46.886336 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:46.886976 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:46.887162 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:46.891516 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:46.891523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:46.891537 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:46.891577 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:46.894632 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:46.895458 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:46.895465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:46.895471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:46.895644 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:46.896133 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:46.896137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:46.896139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:46.896629 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:46.896946 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:46.897486 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 03:04:46.897783 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":331,"threads_started":1}
I20260926 03:04:46.902696 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:46.906781 21824 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8094236d1be4f7e842d568ecbcbf287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260926 03:04:46.907269 21824 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:46.907408 21824 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8094236d1be4f7e842d568ecbcbf287, State: Initialized, Role: FOLLOWER
I20260926 03:04:46.908006 21824 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8094236d1be4f7e842d568ecbcbf287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260926 03:04:46.908242 21824 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:46.908390 21824 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:46.908581 21824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:46.910832 21824 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8094236d1be4f7e842d568ecbcbf287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260926 03:04:46.911316 21824 leader_election.cc:302] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8094236d1be4f7e842d568ecbcbf287; no voters:
I20260926 03:04:46.911984 21824 leader_election.cc:288] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:46.912113 21834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:46.915757 21824 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:46.914839 21834 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [term 1 LEADER]: Becoming Leader. State: Replica: d8094236d1be4f7e842d568ecbcbf287, State: Running, Role: LEADER
I20260926 03:04:46.916386 21834 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8094236d1be4f7e842d568ecbcbf287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } }
I20260926 03:04:46.919323 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8094236d1be4f7e842d568ecbcbf287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8094236d1be4f7e842d568ecbcbf287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } } }
I20260926 03:04:46.919766 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:46.921610 21839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:46.922384 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:46.906742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:47065 (local address 127.0.0.1:40359)
0926 03:04:46.906907 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:46.906911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:46.907477 (+ 566us) server_negotiation.cc:407] Connection header received
0926 03:04:46.907686 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:46.907690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:46.907753 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:46.907854 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:46.909494 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:46.910279 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:46.911559 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:46.912021 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:46.912167 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:46.914661 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:46.914670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:46.914678 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:46.914722 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:46.918145 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:46.919140 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:46.919151 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:46.919156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:46.919365 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:46.919940 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:46.919945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:46.919948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:46.920410 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:46.920785 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:46.921631 (+ 846us) server_negotiation.cc:299] Negotiation successful
0926 03:04:46.921869 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:04:46.923169 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8094236d1be4f7e842d568ecbcbf287. Latest consensus state: current_term: 1 leader_uuid: "d8094236d1be4f7e842d568ecbcbf287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8094236d1be4f7e842d568ecbcbf287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40359 } } }
I20260926 03:04:46.923530 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:46.927975 21839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:46.939028 21839 catalog_manager.cc:1347] Generated new cluster ID: ebfe9c993692438486cfc1b2b4e46823
I20260926 03:04:46.939177 21839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:46.951596 21839 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:46.953166 21839 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:46.965858 21839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8094236d1be4f7e842d568ecbcbf287: Generated new TSK 0
I20260926 03:04:46.966893 21839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:46.981783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21757
I20260926 03:04:46.983161 21819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:47.075687 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21757
W20260926 03:04:47.092868 21742 connection.cc:535] client connection to 127.0.0.1:40359 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40359): Connection reset by peer (error 104)
2026-09-26T03:04:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:47 dist-test-slave-fccw krb5kdc[21857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:47 dist-test-slave-fccw krb5kdc[21857](info): set up 2 sockets
Sep 26 03:04:47 dist-test-slave-fccw krb5kdc[21857](info): commencing operation
krb5kdc: starting...
W20260926 03:04:48.165391 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:48 dist-test-slave-fccw krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:48.411660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:48.738163 21870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:48.738502 21870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:48.738665 21870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:48.746417 21870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:48.746560 21870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:48.746650 21870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:48.746737 21870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:48.746824 21870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:48.756774 21870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:48.758080 21870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:48.759411 21870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:48.770926 21875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:48.771000 21876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:48.771159 21878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:48.771992 21870 server_base.cc:1034] running on GCE node
I20260926 03:04:48.772655 21870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:48.774045 21870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:48.775241 21870 hybrid_clock.cc:648] HybridClock initialized: now 1790391888775205 us; error 23 us; skew 500 ppm
I20260926 03:04:48.775580 21870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:48 dist-test-slave-fccw krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:48.778590 21870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:48.780100 21870 webserver.cc:460] Webserver started at http://127.0.0.1:45695/ using document root <none> and password file <none>
I20260926 03:04:48.780767 21870 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:48.780911 21870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:48.781240 21870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:48.784179 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "1387fa5644ae46c387885e90f4fc8c99"
format_stamp: "Formatted at 2026-09-26 03:04:48 on dist-test-slave-fccw"
server_key: "b9464735682ff9b3461df83a9464a1b2"
server_key_iv: "ba0a0a9372fd01f8296219fd050faaf9"
server_key_version: "encryptionkey@0"
I20260926 03:04:48.784925 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "1387fa5644ae46c387885e90f4fc8c99"
format_stamp: "Formatted at 2026-09-26 03:04:48 on dist-test-slave-fccw"
server_key: "b9464735682ff9b3461df83a9464a1b2"
server_key_iv: "ba0a0a9372fd01f8296219fd050faaf9"
server_key_version: "encryptionkey@0"
I20260926 03:04:48.789822 21870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 03:04:48.793098 21885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:48.794365 21870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:04:48.794536 21870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "1387fa5644ae46c387885e90f4fc8c99"
format_stamp: "Formatted at 2026-09-26 03:04:48 on dist-test-slave-fccw"
server_key: "b9464735682ff9b3461df83a9464a1b2"
server_key_iv: "ba0a0a9372fd01f8296219fd050faaf9"
server_key_version: "encryptionkey@0"
I20260926 03:04:48.794721 21870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:48.818396 21870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:48.822238 21870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:48.822518 21870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:48.840695 21870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42683
I20260926 03:04:48.840724 21936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42683 every 8 connection(s)
I20260926 03:04:48.842032 21870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:48.845584 21937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:48.846028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21870
I20260926 03:04:48.846216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:48.846756 388 external_mini_cluster.cc:1428] Setting key 936c6d1f4205d3996c37d210be4e8b98
I20260926 03:04:48.857767 21937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99: Bootstrap starting.
I20260926 03:04:48.859879 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:48.848618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:60949 (local address 127.0.0.1:42683)
0926 03:04:48.849383 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:48.849401 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:48.849439 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:04:48.851735 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:48.851759 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:48.852143 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:48.852521 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:48.853050 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:48.854322 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:48.855659 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:48.856326 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:48.856521 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:48.857760 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:48.857767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:48.857779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:48.858324 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:48.858409 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:48.858432 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:48.858591 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:48.858847 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 03:04:48.859007 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":366,"threads_started":1}
I20260926 03:04:48.860869 21937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:48.861968 21937 log.cc:824] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:48.863555 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:48.865113 21937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99: No bootstrap required, opened a new log
I20260926 03:04:48.875068 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:48.868464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:60263 (local address 127.0.0.1:42683)
0926 03:04:48.868643 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:48.868655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:48.869174 (+ 519us) server_negotiation.cc:407] Connection header received
0926 03:04:48.869320 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:48.869325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:48.869406 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:48.869510 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:48.870011 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:48.871190 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:48.872475 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:48.873252 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:48.873382 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:48.874389 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:48.874402 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:48.874410 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:48.874429 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:48.874469 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:48.874486 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:48.874596 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:48.874726 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 03:04:48.874773 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 03:04:48.875099 21937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1387fa5644ae46c387885e90f4fc8c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 03:04:48.875648 21937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:48.875778 21937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1387fa5644ae46c387885e90f4fc8c99, State: Initialized, Role: FOLLOWER
I20260926 03:04:48.876401 21937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1387fa5644ae46c387885e90f4fc8c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 03:04:48.876682 21937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:48.876832 21937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:48.877030 21937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:48.879122 21937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1387fa5644ae46c387885e90f4fc8c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 03:04:48.879607 21937 leader_election.cc:302] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1387fa5644ae46c387885e90f4fc8c99; no voters:
I20260926 03:04:48.880170 21937 leader_election.cc:288] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:48.880313 21947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:48.881102 21947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [term 1 LEADER]: Becoming Leader. State: Replica: 1387fa5644ae46c387885e90f4fc8c99, State: Running, Role: LEADER
I20260926 03:04:48.881644 21947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1387fa5644ae46c387885e90f4fc8c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } }
I20260926 03:04:48.882315 21937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:48.884608 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1387fa5644ae46c387885e90f4fc8c99. Latest consensus state: current_term: 1 leader_uuid: "1387fa5644ae46c387885e90f4fc8c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1387fa5644ae46c387885e90f4fc8c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260926 03:04:48.884756 21948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1387fa5644ae46c387885e90f4fc8c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1387fa5644ae46c387885e90f4fc8c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42683 } } }
I20260926 03:04:48.885102 21948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:48.885103 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:48.886794 21955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:48.891227 21955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:48.899914 21955 catalog_manager.cc:1347] Generated new cluster ID: 5f3f0969f3e14747acf582508496a2bb
I20260926 03:04:48.900027 21955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:48.915798 21955 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:48.917194 21955 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:48.927268 21955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1387fa5644ae46c387885e90f4fc8c99: Generated new TSK 0
I20260926 03:04:48.928390 21955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:48.941566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21870
W20260926 03:04:48.942586 21932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.114:60949
W20260926 03:04:48.943349 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21870: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:48.943493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21870
W20260926 03:04:48.960072 21855 connection.cc:535] client connection to 127.0.0.1:42683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42683): Connection reset by peer (error 104)
2026-09-26T03:04:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:49 dist-test-slave-fccw krb5kdc[21968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:49 dist-test-slave-fccw krb5kdc[21968](info): set up 2 sockets
Sep 26 03:04:49 dist-test-slave-fccw krb5kdc[21968](info): commencing operation
krb5kdc: starting...
W20260926 03:04:50.033443 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:50 dist-test-slave-fccw krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 03:04:50.272675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:50.593735 21981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:50.594116 21981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:50.594218 21981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:50.601286 21981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:50.601392 21981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:50.601444 21981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:50.601501 21981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:50.601543 21981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:50.611107 21981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:50.612339 21981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:50.613643 21981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:50.625221 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:50.625258 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:50.625303 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:50.626062 21981 server_base.cc:1034] running on GCE node
I20260926 03:04:50.626801 21981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:50.628129 21981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:50.629347 21981 hybrid_clock.cc:648] HybridClock initialized: now 1790391890629303 us; error 57 us; skew 500 ppm
I20260926 03:04:50.629693 21981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:50 dist-test-slave-fccw krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:50.637265 21981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 03:04:50.638988 21981 webserver.cc:460] Webserver started at http://127.0.0.1:38595/ using document root <none> and password file <none>
I20260926 03:04:50.639647 21981 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:50.639776 21981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:50.640075 21981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:50.642891 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "527e11b935a54351a53a432f1916b8aa"
format_stamp: "Formatted at 2026-09-26 03:04:50 on dist-test-slave-fccw"
server_key: "3dca660c79f65b370df35543b888abbd"
server_key_iv: "68d842fb151ef03b2c5085742a58eeba"
server_key_version: "encryptionkey@0"
I20260926 03:04:50.643608 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "527e11b935a54351a53a432f1916b8aa"
format_stamp: "Formatted at 2026-09-26 03:04:50 on dist-test-slave-fccw"
server_key: "3dca660c79f65b370df35543b888abbd"
server_key_iv: "68d842fb151ef03b2c5085742a58eeba"
server_key_version: "encryptionkey@0"
I20260926 03:04:50.648315 21981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 03:04:50.651506 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:50.652709 21981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 03:04:50.652889 21981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "527e11b935a54351a53a432f1916b8aa"
format_stamp: "Formatted at 2026-09-26 03:04:50 on dist-test-slave-fccw"
server_key: "3dca660c79f65b370df35543b888abbd"
server_key_iv: "68d842fb151ef03b2c5085742a58eeba"
server_key_version: "encryptionkey@0"
I20260926 03:04:50.653132 21981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:50.658481 21981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:50.662160 21981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:50.662446 21981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:50.679944 21981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46095
I20260926 03:04:50.680016 22047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46095 every 8 connection(s)
I20260926 03:04:50.681442 21981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:50.684787 22048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:50.685539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 21981
I20260926 03:04:50.685725 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:50.686228 388 external_mini_cluster.cc:1428] Setting key 17e04c2653dc711d27d97f6992a28197
I20260926 03:04:50.693692 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:50.688037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:45697 (local address 127.0.0.1:46095)
0926 03:04:50.688792 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:50.688802 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:50.688849 (+ 47us) server_negotiation.cc:407] Connection header received
0926 03:04:50.690552 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:50.690573 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:50.690884 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:50.691192 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:50.692157 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:50.692163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:50.692174 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:50.692187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:50.692243 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:50.692261 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:50.692419 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:50.692553 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 03:04:50.692738 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":351,"threads_started":1}
I20260926 03:04:50.697341 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:50.697803 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa: Bootstrap starting.
I20260926 03:04:50.700659 22048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:50.701797 22048 log.cc:824] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:50.704027 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:50.701589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.114:35377 (local address 127.0.0.1:46095)
0926 03:04:50.701790 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:50.701798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:50.702272 (+ 474us) server_negotiation.cc:407] Connection header received
0926 03:04:50.702420 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:50.702423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:50.702462 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:50.702538 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:50.703425 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:50.703430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:50.703447 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:50.703466 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:50.703521 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:50.703539 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:50.703651 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:50.703715 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 03:04:50.703767 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:04:50.704855 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa: No bootstrap required, opened a new log
I20260926 03:04:50.712260 22048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527e11b935a54351a53a432f1916b8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46095 } }
I20260926 03:04:50.712560 22048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:50.712630 22048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527e11b935a54351a53a432f1916b8aa, State: Initialized, Role: FOLLOWER
I20260926 03:04:50.713305 22048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527e11b935a54351a53a432f1916b8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46095 } }
I20260926 03:04:50.713583 22048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:50.713724 22048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:50.713853 22048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:50.715695 22048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527e11b935a54351a53a432f1916b8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46095 } }
I20260926 03:04:50.716087 22048 leader_election.cc:302] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 527e11b935a54351a53a432f1916b8aa; no voters:
I20260926 03:04:50.716558 22048 leader_election.cc:288] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:50.716702 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:50.717425 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [term 1 LEADER]: Becoming Leader. State: Replica: 527e11b935a54351a53a432f1916b8aa, State: Running, Role: LEADER
I20260926 03:04:50.717903 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527e11b935a54351a53a432f1916b8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46095 } }
I20260926 03:04:50.718783 22048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:50.721223 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "527e11b935a54351a53a432f1916b8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527e11b935a54351a53a432f1916b8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46095 } } }
I20260926 03:04:50.721637 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:50.721186 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527e11b935a54351a53a432f1916b8aa. Latest consensus state: current_term: 1 leader_uuid: "527e11b935a54351a53a432f1916b8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "527e11b935a54351a53a432f1916b8aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46095 } } }
I20260926 03:04:50.722177 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:50.723280 22065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:50.728298 22065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:50.737143 22065 catalog_manager.cc:1347] Generated new cluster ID: 6633f54f964d4818a2ebea11dcdeba78
I20260926 03:04:50.737248 22065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:50.763266 22065 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:50.763695 22065 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:50.770757 22065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 527e11b935a54351a53a432f1916b8aa: Generated new TSK 0
I20260926 03:04:50.771479 22065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:50.782521 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21981
W20260926 03:04:50.783542 22043 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.114:45697
W20260926 03:04:50.784186 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 21981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:50.784327 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 21981
2026-09-26T03:04:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1840 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:50 dist-test-slave-fccw krb5kdc[22079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:50 dist-test-slave-fccw krb5kdc[22079](info): set up 2 sockets
Sep 26 03:04:50 dist-test-slave-fccw krb5kdc[22079](info): commencing operation
krb5kdc: starting...
W20260926 03:04:51.857318 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:52 dist-test-slave-fccw krb5kdc[22079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:52.095889 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38853
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:52.412921 22092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:52.413278 22092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:52.413437 22092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:52.420581 22092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:52.420748 22092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:52.420867 22092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:52.420980 22092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:52.421096 22092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:52.430859 22092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:52.432127 22092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:52.433458 22092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:52.444943 22097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:52.444911 22098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:52.444887 22100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:52.445734 22092 server_base.cc:1034] running on GCE node
I20260926 03:04:52.446501 22092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:52.447819 22092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:52.449079 22092 hybrid_clock.cc:648] HybridClock initialized: now 1790391892449031 us; error 44 us; skew 500 ppm
I20260926 03:04:52.449404 22092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:52 dist-test-slave-fccw krb5kdc[22079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:52.452276 22092 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:04:52.453694 22092 webserver.cc:460] Webserver started at http://127.0.97.62:43737/ using document root <none> and password file <none>
I20260926 03:04:52.454411 22092 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:52.454561 22092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:52.454852 22092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:52.457764 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "c356de0ac1484aa393d64b1ce1b5967c"
format_stamp: "Formatted at 2026-09-26 03:04:52 on dist-test-slave-fccw"
server_key: "fa613ad3fb1dcb7ecd8cb82a676505c4"
server_key_iv: "e7bcc4cf4ece707cbeb823086610abce"
server_key_version: "encryptionkey@0"
I20260926 03:04:52.458540 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "c356de0ac1484aa393d64b1ce1b5967c"
format_stamp: "Formatted at 2026-09-26 03:04:52 on dist-test-slave-fccw"
server_key: "fa613ad3fb1dcb7ecd8cb82a676505c4"
server_key_iv: "e7bcc4cf4ece707cbeb823086610abce"
server_key_version: "encryptionkey@0"
I20260926 03:04:52.463467 22092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 03:04:52.466794 22107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:52.468005 22092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:04:52.468199 22092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "c356de0ac1484aa393d64b1ce1b5967c"
format_stamp: "Formatted at 2026-09-26 03:04:52 on dist-test-slave-fccw"
server_key: "fa613ad3fb1dcb7ecd8cb82a676505c4"
server_key_iv: "e7bcc4cf4ece707cbeb823086610abce"
server_key_version: "encryptionkey@0"
I20260926 03:04:52.468395 22092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:52.482707 22092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:52.486754 22092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:52.487049 22092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:52.505141 22092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38853
I20260926 03:04:52.505158 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38853 every 8 connection(s)
I20260926 03:04:52.506440 22092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:52.507627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22092
I20260926 03:04:52.507798 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:52.508311 388 external_mini_cluster.cc:1428] Setting key d04b10f9d137e154e7a692004d4f2fee
I20260926 03:04:52.510900 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:52.521389 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c: Bootstrap starting.
Sep 26 03:04:52 dist-test-slave-fccw krb5kdc[22079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:52.525070 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:52.526151 22159 log.cc:824] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:52.529440 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c: No bootstrap required, opened a new log
I20260926 03:04:52.530364 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:52.510103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44255 (local address 127.0.97.62:38853)
0926 03:04:52.510836 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:52.510848 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:52.510897 (+ 49us) server_negotiation.cc:407] Connection header received
0926 03:04:52.512577 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:52.512598 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:52.512959 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:52.513450 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:52.514827 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:52.516154 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:52.517522 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:52.518170 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:52.518393 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:52.522933 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:52.522940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:52.522953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:52.522995 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:52.526025 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:52.527001 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:52.527010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:52.527020 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:52.527260 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:52.527747 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:52.527750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:52.527753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:52.528221 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:52.528471 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:52.528959 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 03:04:52.529323 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":325,"threads_started":1}
I20260926 03:04:52.534080 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:52.538759 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c356de0ac1484aa393d64b1ce1b5967c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38853 } }
I20260926 03:04:52.539178 22159 raft_consensus.cc:374] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:52.539350 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c356de0ac1484aa393d64b1ce1b5967c, State: Initialized, Role: FOLLOWER
I20260926 03:04:52.539979 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c356de0ac1484aa393d64b1ce1b5967c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38853 } }
I20260926 03:04:52.540222 22159 raft_consensus.cc:388] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:52.540381 22159 raft_consensus.cc:482] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:52.540555 22159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:52.542775 22159 raft_consensus.cc:504] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c356de0ac1484aa393d64b1ce1b5967c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38853 } }
I20260926 03:04:52.543239 22159 leader_election.cc:302] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c356de0ac1484aa393d64b1ce1b5967c; no voters:
I20260926 03:04:52.543872 22159 leader_election.cc:288] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:52.543980 22169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:52.545259 22169 raft_consensus.cc:686] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [term 1 LEADER]: Becoming Leader. State: Replica: c356de0ac1484aa393d64b1ce1b5967c, State: Running, Role: LEADER
I20260926 03:04:52.545811 22169 consensus_queue.cc:237] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c356de0ac1484aa393d64b1ce1b5967c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38853 } }
I20260926 03:04:52.546346 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:52.549118 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c356de0ac1484aa393d64b1ce1b5967c. Latest consensus state: current_term: 1 leader_uuid: "c356de0ac1484aa393d64b1ce1b5967c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c356de0ac1484aa393d64b1ce1b5967c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38853 } } }
I20260926 03:04:52.549569 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:52.550945 22170 sys_catalog.cc:455] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c356de0ac1484aa393d64b1ce1b5967c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c356de0ac1484aa393d64b1ce1b5967c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38853 } } }
I20260926 03:04:52.551352 22170 sys_catalog.cc:458] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:52.552875 22175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:52.555855 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:52.539083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42739 (local address 127.0.97.62:38853)
0926 03:04:52.539263 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:52.539271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:52.539736 (+ 465us) server_negotiation.cc:407] Connection header received
0926 03:04:52.539919 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:52.539925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:52.539999 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:52.540102 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:52.541665 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:52.542451 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:52.543794 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:52.544686 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:52.544802 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:52.547872 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:52.547880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:52.547887 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:52.547937 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:52.552055 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:52.552953 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:52.552960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:52.552966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:52.553204 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:52.553798 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:52.553805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:52.553808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:52.554191 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:52.554522 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:52.555094 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 03:04:52.555326 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:04:52.559789 22175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:52.569080 22175 catalog_manager.cc:1347] Generated new cluster ID: 150ac90bcca8434e9334aac9ce66188c
I20260926 03:04:52.569196 22175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:52.582819 22175 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:52.584576 22175 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:52.601025 22175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c356de0ac1484aa393d64b1ce1b5967c: Generated new TSK 0
I20260926 03:04:52.602075 22175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:52.618428 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22092
I20260926 03:04:52.619944 22154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:52.710594 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22092
W20260926 03:04:52.727854 22077 connection.cc:535] client connection to 127.0.97.62:38853 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:38853): Connection reset by peer (error 104)
2026-09-26T03:04:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:52 dist-test-slave-fccw krb5kdc[22191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:52 dist-test-slave-fccw krb5kdc[22191](info): set up 2 sockets
Sep 26 03:04:52 dist-test-slave-fccw krb5kdc[22191](info): commencing operation
krb5kdc: starting...
W20260926 03:04:53.805469 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:53 dist-test-slave-fccw krb5kdc[22191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:54.039938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33295
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:54.363296 22204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:54.363577 22204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:54.363667 22204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:54.370661 22204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:54.370818 22204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:54.370911 22204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:54.370975 22204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:54.371016 22204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:54.382958 22204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:54.384580 22204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:54.385861 22204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:54.397539 22209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:54.397629 22212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:54.397588 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:54.398239 22204 server_base.cc:1034] running on GCE node
I20260926 03:04:54.398995 22204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:54.400390 22204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:54.401613 22204 hybrid_clock.cc:648] HybridClock initialized: now 1790391894401568 us; error 58 us; skew 500 ppm
I20260926 03:04:54.401991 22204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:54 dist-test-slave-fccw krb5kdc[22191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:54.404944 22204 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:04:54.406435 22204 webserver.cc:460] Webserver started at http://127.0.97.62:40749/ using document root <none> and password file <none>
I20260926 03:04:54.407055 22204 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:54.407150 22204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:54.407411 22204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:54.410300 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "65702cb2a9fa440da2a63e1ca8c94065"
format_stamp: "Formatted at 2026-09-26 03:04:54 on dist-test-slave-fccw"
server_key: "5b021baa637780233077c415b3cc8e34"
server_key_iv: "f21e5ec65b10b046ef779068e91bb57b"
server_key_version: "encryptionkey@0"
I20260926 03:04:54.410985 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "65702cb2a9fa440da2a63e1ca8c94065"
format_stamp: "Formatted at 2026-09-26 03:04:54 on dist-test-slave-fccw"
server_key: "5b021baa637780233077c415b3cc8e34"
server_key_iv: "f21e5ec65b10b046ef779068e91bb57b"
server_key_version: "encryptionkey@0"
I20260926 03:04:54.416007 22204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:04:54.419286 22219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:54.420485 22204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 03:04:54.420612 22204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "65702cb2a9fa440da2a63e1ca8c94065"
format_stamp: "Formatted at 2026-09-26 03:04:54 on dist-test-slave-fccw"
server_key: "5b021baa637780233077c415b3cc8e34"
server_key_iv: "f21e5ec65b10b046ef779068e91bb57b"
server_key_version: "encryptionkey@0"
I20260926 03:04:54.420759 22204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:54.435199 22204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:54.438941 22204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:54.439153 22204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:54.457089 22204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33295
I20260926 03:04:54.457199 22270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33295 every 8 connection(s)
I20260926 03:04:54.458463 22204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:54.462127 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:54.463362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22204
I20260926 03:04:54.463521 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:54.463953 388 external_mini_cluster.cc:1428] Setting key 71283180495daa091a5dee3f99e6a41e
I20260926 03:04:54.475967 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065: Bootstrap starting.
I20260926 03:04:54.478410 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:54.465769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44413 (local address 127.0.97.62:33295)
0926 03:04:54.466525 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:54.466541 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:54.466587 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:04:54.468462 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:54.468488 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:54.469019 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:54.469457 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:54.469924 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:54.471750 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:54.473080 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:54.474104 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:54.474264 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:54.475640 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:54.475650 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:54.475667 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:54.476443 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:54.476556 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:54.476590 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:54.476731 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:54.476969 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 03:04:54.477147 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":317,"threads_started":1}
I20260926 03:04:54.479875 22271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:54.481071 22271 log.cc:824] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:54.482219 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:54.484135 22271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065: No bootstrap required, opened a new log
I20260926 03:04:54.493497 22271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65702cb2a9fa440da2a63e1ca8c94065" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33295 } }
I20260926 03:04:54.493885 22271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:54.494009 22271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65702cb2a9fa440da2a63e1ca8c94065, State: Initialized, Role: FOLLOWER
I20260926 03:04:54.494696 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:54.487309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35755 (local address 127.0.97.62:33295)
0926 03:04:54.487577 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:54.487587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:54.488225 (+ 638us) server_negotiation.cc:407] Connection header received
0926 03:04:54.488372 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:54.488378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:54.488445 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:54.488540 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:54.489109 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:54.490313 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:54.491832 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:54.492528 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:54.492663 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:54.493944 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:54.493950 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:54.493953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:54.493970 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:54.494024 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:54.494040 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:54.494171 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:54.494378 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 03:04:54.494437 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 03:04:54.494642 22271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65702cb2a9fa440da2a63e1ca8c94065" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33295 } }
I20260926 03:04:54.494966 22271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:54.495118 22271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:54.495319 22271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:54.497742 22271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65702cb2a9fa440da2a63e1ca8c94065" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33295 } }
I20260926 03:04:54.498160 22271 leader_election.cc:302] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65702cb2a9fa440da2a63e1ca8c94065; no voters:
I20260926 03:04:54.498663 22271 leader_election.cc:288] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:54.498805 22281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:54.499492 22281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [term 1 LEADER]: Becoming Leader. State: Replica: 65702cb2a9fa440da2a63e1ca8c94065, State: Running, Role: LEADER
I20260926 03:04:54.500063 22281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65702cb2a9fa440da2a63e1ca8c94065" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33295 } }
I20260926 03:04:54.500685 22271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:54.502688 22283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65702cb2a9fa440da2a63e1ca8c94065. Latest consensus state: current_term: 1 leader_uuid: "65702cb2a9fa440da2a63e1ca8c94065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65702cb2a9fa440da2a63e1ca8c94065" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33295 } } }
I20260926 03:04:54.503062 22283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:54.505750 22282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65702cb2a9fa440da2a63e1ca8c94065" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65702cb2a9fa440da2a63e1ca8c94065" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33295 } } }
I20260926 03:04:54.506088 22282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:54.507755 22287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:54.511651 22287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:54.520287 22287 catalog_manager.cc:1347] Generated new cluster ID: 3b97983015734c679fc21470ddf37541
I20260926 03:04:54.520412 22287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:54.543285 22287 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:54.544603 22287 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:54.551517 22287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65702cb2a9fa440da2a63e1ca8c94065: Generated new TSK 0
I20260926 03:04:54.552294 22287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:54.563103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22204
W20260926 03:04:54.564361 22266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44413
W20260926 03:04:54.565143 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:54.565289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22204
W20260926 03:04:54.581892 22189 connection.cc:535] client connection to 127.0.97.62:33295 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:33295): Connection reset by peer (error 104)
2026-09-26T03:04:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1853 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:54 dist-test-slave-fccw krb5kdc[22302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:54 dist-test-slave-fccw krb5kdc[22302](info): set up 2 sockets
Sep 26 03:04:54 dist-test-slave-fccw krb5kdc[22302](info): commencing operation
krb5kdc: starting...
W20260926 03:04:55.645473 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:55 dist-test-slave-fccw krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:55.884254 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42031
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:56.229172 22315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:56.229593 22315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:56.229760 22315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:56.237733 22315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:56.237877 22315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:56.237982 22315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:56.238070 22315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:56.238154 22315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:56.248258 22315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:56.249672 22315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:56.250972 22315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:56.262621 22320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:56.262835 22323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:56.262913 22321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:56.263541 22315 server_base.cc:1034] running on GCE node
I20260926 03:04:56.264242 22315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:56.265666 22315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:56.266894 22315 hybrid_clock.cc:648] HybridClock initialized: now 1790391896266857 us; error 35 us; skew 500 ppm
I20260926 03:04:56.267263 22315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:56 dist-test-slave-fccw krb5kdc[22302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:56.270427 22315 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:04:56.271895 22315 webserver.cc:460] Webserver started at http://127.0.97.62:38363/ using document root <none> and password file <none>
I20260926 03:04:56.272555 22315 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:56.272692 22315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:56.272986 22315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:56.276090 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "4ca492e19cc549ed8d8e82c421bd826b"
format_stamp: "Formatted at 2026-09-26 03:04:56 on dist-test-slave-fccw"
server_key: "fe4a6555aff905e930bea5a7a05160b7"
server_key_iv: "bbaea620d601a95aa74002df951d298d"
server_key_version: "encryptionkey@0"
I20260926 03:04:56.276836 22315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "4ca492e19cc549ed8d8e82c421bd826b"
format_stamp: "Formatted at 2026-09-26 03:04:56 on dist-test-slave-fccw"
server_key: "fe4a6555aff905e930bea5a7a05160b7"
server_key_iv: "bbaea620d601a95aa74002df951d298d"
server_key_version: "encryptionkey@0"
I20260926 03:04:56.281916 22315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 03:04:56.285238 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:56.286536 22315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 03:04:56.286716 22315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "4ca492e19cc549ed8d8e82c421bd826b"
format_stamp: "Formatted at 2026-09-26 03:04:56 on dist-test-slave-fccw"
server_key: "fe4a6555aff905e930bea5a7a05160b7"
server_key_iv: "bbaea620d601a95aa74002df951d298d"
server_key_version: "encryptionkey@0"
I20260926 03:04:56.286912 22315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:56.292423 22315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:56.296077 22315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:56.296360 22315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:56.315086 22315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42031
I20260926 03:04:56.315128 22381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42031 every 8 connection(s)
I20260926 03:04:56.316407 22315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:56.318387 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22315
I20260926 03:04:56.318568 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:56.319087 388 external_mini_cluster.cc:1428] Setting key d4604f7f85d32fc31a948f8d8a7b4a9d
I20260926 03:04:56.320585 22382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:56.326496 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:56.320798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41415 (local address 127.0.97.62:42031)
0926 03:04:56.321524 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:56.321535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:56.321573 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:04:56.323122 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:56.323143 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:56.323468 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:56.323873 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:56.324867 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:56.324873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:56.324885 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:56.324898 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:56.324969 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:56.324987 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:56.325254 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:56.325390 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 03:04:56.325578 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":333,"threads_started":1}
I20260926 03:04:56.329924 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:56.332120 22382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b: Bootstrap starting.
I20260926 03:04:56.335485 22382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:56.336851 22382 log.cc:824] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:56.337158 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:56.334727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37151 (local address 127.0.97.62:42031)
0926 03:04:56.334916 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:56.334921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:56.335179 (+ 258us) server_negotiation.cc:407] Connection header received
0926 03:04:56.335399 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:56.335402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:56.335446 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:56.335534 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:56.336404 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:56.336408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:56.336411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:04:56.336424 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:56.336463 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:04:56.336476 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:56.336561 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:56.336719 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 03:04:56.336762 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:04:56.339802 22382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b: No bootstrap required, opened a new log
I20260926 03:04:56.347175 22382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca492e19cc549ed8d8e82c421bd826b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42031 } }
I20260926 03:04:56.347563 22382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:56.347681 22382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca492e19cc549ed8d8e82c421bd826b, State: Initialized, Role: FOLLOWER
I20260926 03:04:56.348309 22382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca492e19cc549ed8d8e82c421bd826b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42031 } }
I20260926 03:04:56.348594 22382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:56.348768 22382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:56.348948 22382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:56.351089 22382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca492e19cc549ed8d8e82c421bd826b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42031 } }
I20260926 03:04:56.351588 22382 leader_election.cc:302] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ca492e19cc549ed8d8e82c421bd826b; no voters:
I20260926 03:04:56.352202 22382 leader_election.cc:288] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:56.352453 22392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:56.353188 22392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [term 1 LEADER]: Becoming Leader. State: Replica: 4ca492e19cc549ed8d8e82c421bd826b, State: Running, Role: LEADER
I20260926 03:04:56.353731 22392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca492e19cc549ed8d8e82c421bd826b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42031 } }
I20260926 03:04:56.354328 22382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:56.356597 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ca492e19cc549ed8d8e82c421bd826b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca492e19cc549ed8d8e82c421bd826b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42031 } } }
I20260926 03:04:56.357079 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:56.358729 22399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:56.358758 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ca492e19cc549ed8d8e82c421bd826b. Latest consensus state: current_term: 1 leader_uuid: "4ca492e19cc549ed8d8e82c421bd826b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca492e19cc549ed8d8e82c421bd826b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42031 } } }
I20260926 03:04:56.359134 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:56.364115 22399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:56.373915 22399 catalog_manager.cc:1347] Generated new cluster ID: d7ac03d3d38345fe9b3c86f2e09ea5c2
I20260926 03:04:56.374037 22399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:56.410002 22399 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:56.410532 22399 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:56.430204 22399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ca492e19cc549ed8d8e82c421bd826b: Generated new TSK 0
I20260926 03:04:56.431285 22399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:56.437021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22315
W20260926 03:04:56.438045 22377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41415
W20260926 03:04:56.438740 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22315: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:04:56.438891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22315
2026-09-26T03:04:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1873 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:56 dist-test-slave-fccw krb5kdc[22413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:56 dist-test-slave-fccw krb5kdc[22413](info): set up 2 sockets
Sep 26 03:04:56 dist-test-slave-fccw krb5kdc[22413](info): commencing operation
krb5kdc: starting...
W20260926 03:04:57.537391 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:57 dist-test-slave-fccw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:57.771467 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37799
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:04:58.092314 22426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:04:58.092609 22426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:04:58.092783 22426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:04:58.099848 22426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:04:58.099951 22426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:04:58.099996 22426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:04:58.100100 22426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:04:58.100184 22426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:04:58.109910 22426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:04:58.111100 22426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:04:58.112475 22426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:04:58.124183 22434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:58.124258 22432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:04:58.124578 22431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:04:58.125401 22426 server_base.cc:1034] running on GCE node
I20260926 03:04:58.126123 22426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:04:58.127476 22426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:04:58.128703 22426 hybrid_clock.cc:648] HybridClock initialized: now 1790391898128665 us; error 33 us; skew 500 ppm
I20260926 03:04:58.129106 22426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:04:58 dist-test-slave-fccw krb5kdc[22413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:04:58.131958 22426 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:04:58.133365 22426 webserver.cc:460] Webserver started at http://127.0.97.62:34893/ using document root <none> and password file <none>
I20260926 03:04:58.133965 22426 fs_manager.cc:359] Metadata directory not provided
I20260926 03:04:58.134057 22426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:04:58.134308 22426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:04:58.137099 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "b13a5aeca1b3470bb04ce895458963f0"
format_stamp: "Formatted at 2026-09-26 03:04:58 on dist-test-slave-fccw"
server_key: "23392b382f821fbdac68cc56f9aae27c"
server_key_iv: "3d62b311952cc2fff271bc7ce2dc7bd4"
server_key_version: "encryptionkey@0"
I20260926 03:04:58.137797 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "b13a5aeca1b3470bb04ce895458963f0"
format_stamp: "Formatted at 2026-09-26 03:04:58 on dist-test-slave-fccw"
server_key: "23392b382f821fbdac68cc56f9aae27c"
server_key_iv: "3d62b311952cc2fff271bc7ce2dc7bd4"
server_key_version: "encryptionkey@0"
I20260926 03:04:58.142570 22426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 03:04:58.145707 22441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:04:58.146903 22426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 03:04:58.147029 22426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "b13a5aeca1b3470bb04ce895458963f0"
format_stamp: "Formatted at 2026-09-26 03:04:58 on dist-test-slave-fccw"
server_key: "23392b382f821fbdac68cc56f9aae27c"
server_key_iv: "3d62b311952cc2fff271bc7ce2dc7bd4"
server_key_version: "encryptionkey@0"
I20260926 03:04:58.147171 22426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:04:58.161748 22426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:04:58.165463 22426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:04:58.165690 22426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:04:58.183787 22426 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37799
I20260926 03:04:58.183825 22492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37799 every 8 connection(s)
I20260926 03:04:58.185149 22426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:04:58.188562 22493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:04:58.193723 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22426
I20260926 03:04:58.193902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:04:58.194391 388 external_mini_cluster.cc:1428] Setting key 0913011205a835978642e67cd380c856
I20260926 03:04:58.199687 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0: Bootstrap starting.
I20260926 03:04:58.203231 22493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0: Neither blocks nor log segments found. Creating new log.
I20260926 03:04:58.204430 22493 log.cc:824] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:04:58.207239 22493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0: No bootstrap required, opened a new log
Sep 26 03:04:58 dist-test-slave-fccw krb5kdc[22413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:04:58.214805 22493 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13a5aeca1b3470bb04ce895458963f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37799 } }
I20260926 03:04:58.215160 22493 raft_consensus.cc:374] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:04:58.215291 22493 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13a5aeca1b3470bb04ce895458963f0, State: Initialized, Role: FOLLOWER
I20260926 03:04:58.215929 22493 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13a5aeca1b3470bb04ce895458963f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37799 } }
I20260926 03:04:58.216182 22493 raft_consensus.cc:388] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:04:58.216339 22493 raft_consensus.cc:482] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:04:58.216521 22493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:04:58.216868 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:58.196075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51209 (local address 127.0.97.62:37799)
0926 03:04:58.196921 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:58.196935 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:58.196976 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:04:58.198783 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:58.198803 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:58.199167 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:58.199472 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:58.201192 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:58.202464 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:58.204263 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:58.205112 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:58.205312 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:58.209527 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:58.209534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:58.209547 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:58.209578 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:58.212659 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:58.213481 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:58.213494 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:58.213506 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:58.213664 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:58.214056 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:58.214059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:58.214062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:58.214547 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:58.214841 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:58.215281 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 03:04:58.215747 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":432,"threads_started":1}
I20260926 03:04:58.219357 22493 raft_consensus.cc:504] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13a5aeca1b3470bb04ce895458963f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37799 } }
I20260926 03:04:58.219941 22493 leader_election.cc:302] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b13a5aeca1b3470bb04ce895458963f0; no voters:
I20260926 03:04:58.220592 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:04:58.220620 22493 leader_election.cc:288] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:04:58.220741 22498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:04:58.222719 22493 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:04:58.227787 22498 raft_consensus.cc:686] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [term 1 LEADER]: Becoming Leader. State: Replica: b13a5aeca1b3470bb04ce895458963f0, State: Running, Role: LEADER
I20260926 03:04:58.228544 22498 consensus_queue.cc:237] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13a5aeca1b3470bb04ce895458963f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37799 } }
I20260926 03:04:58.231945 22499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b13a5aeca1b3470bb04ce895458963f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13a5aeca1b3470bb04ce895458963f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37799 } } }
I20260926 03:04:58.232405 22499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:58.241652 22500 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b13a5aeca1b3470bb04ce895458963f0. Latest consensus state: current_term: 1 leader_uuid: "b13a5aeca1b3470bb04ce895458963f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13a5aeca1b3470bb04ce895458963f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37799 } } }
I20260926 03:04:58.242034 22500 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:04:58.242560 22511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:04:58.248688 22511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:04:58.255980 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:04:58.226606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54455 (local address 127.0.97.62:37799)
0926 03:04:58.226834 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:04:58.226843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:04:58.238610 (+ 11767us) server_negotiation.cc:407] Connection header received
0926 03:04:58.239137 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:04:58.239146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:04:58.239245 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:04:58.239333 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:04:58.241864 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:58.243080 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:58.245030 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:04:58.245741 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:04:58.245858 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:04:58.249116 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:04:58.249123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:04:58.249128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:04:58.249164 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:04:58.253158 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:58.253931 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:58.253936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:58.253940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:58.254070 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:04:58.254451 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:04:58.254458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:04:58.254461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:04:58.254787 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:04:58.255074 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 03:04:58.255376 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 03:04:58.255599 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 03:04:58.259076 22511 catalog_manager.cc:1347] Generated new cluster ID: 269ba9f21664414b9f7f44620d7cbb6c
I20260926 03:04:58.259181 22511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:04:58.273162 22511 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:04:58.274446 22511 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:04:58.283257 22511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b13a5aeca1b3470bb04ce895458963f0: Generated new TSK 0
I20260926 03:04:58.283999 22511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:04:58.289923 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22426
I20260926 03:04:58.291059 22488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:04:58.381174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22426
2026-09-26T03:04:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:04:58 dist-test-slave-fccw krb5kdc[22526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:04:58 dist-test-slave-fccw krb5kdc[22526](info): set up 2 sockets
Sep 26 03:04:58 dist-test-slave-fccw krb5kdc[22526](info): commencing operation
krb5kdc: starting...
W20260926 03:04:59.485446 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:04:59 dist-test-slave-fccw krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:04:59.723950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33009
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:00.045915 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:00.046273 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:00.046422 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:00.053526 22539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:00.053670 22539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:00.053761 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:00.053864 22539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:00.053946 22539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:00.063688 22539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:00.065450 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:00.067061 22539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:00.078516 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:00.078539 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:00.079118 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:00.079416 22539 server_base.cc:1034] running on GCE node
I20260926 03:05:00.080156 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:00.081501 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:00.082721 22539 hybrid_clock.cc:648] HybridClock initialized: now 1790391900082688 us; error 42 us; skew 500 ppm
I20260926 03:05:00.083062 22539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:00 dist-test-slave-fccw krb5kdc[22526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:00.086195 22539 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:05:00.087555 22539 webserver.cc:460] Webserver started at http://127.0.97.62:43563/ using document root <none> and password file <none>
I20260926 03:05:00.088227 22539 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:00.088384 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:00.088676 22539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:00.091580 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "1bde437aeb694c36ac7ec2c8bbdb8257"
format_stamp: "Formatted at 2026-09-26 03:05:00 on dist-test-slave-fccw"
server_key: "f65d9dc23dbb563135c3dbeef66ddec8"
server_key_iv: "31d464d8688f168b0e5f30c3b0a0c6a5"
server_key_version: "encryptionkey@0"
I20260926 03:05:00.092343 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "1bde437aeb694c36ac7ec2c8bbdb8257"
format_stamp: "Formatted at 2026-09-26 03:05:00 on dist-test-slave-fccw"
server_key: "f65d9dc23dbb563135c3dbeef66ddec8"
server_key_iv: "31d464d8688f168b0e5f30c3b0a0c6a5"
server_key_version: "encryptionkey@0"
I20260926 03:05:00.097177 22539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260926 03:05:00.100399 22554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:00.101745 22539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 03:05:00.101929 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "1bde437aeb694c36ac7ec2c8bbdb8257"
format_stamp: "Formatted at 2026-09-26 03:05:00 on dist-test-slave-fccw"
server_key: "f65d9dc23dbb563135c3dbeef66ddec8"
server_key_iv: "31d464d8688f168b0e5f30c3b0a0c6a5"
server_key_version: "encryptionkey@0"
I20260926 03:05:00.102111 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:00.117439 22539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:00.121670 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:00.121943 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:00.138942 22539 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33009
I20260926 03:05:00.138974 22605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33009 every 8 connection(s)
I20260926 03:05:00.140203 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:05:00.143745 22606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:00.144840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22539
I20260926 03:05:00.145035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:05:00.145442 388 external_mini_cluster.cc:1428] Setting key dc77b7e817917c1b1fe9f1c4dc47f4e2
I20260926 03:05:00.162345 22606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257: Bootstrap starting.
I20260926 03:05:00.163722 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:00.147199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37349 (local address 127.0.97.62:33009)
0926 03:05:00.148315 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:00.148341 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:00.148386 (+ 45us) server_negotiation.cc:407] Connection header received
0926 03:05:00.154803 (+ 6417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:00.154833 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:00.155418 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:00.155836 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:00.156406 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:00.157773 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:00.159622 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:00.160257 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:00.160390 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:00.161593 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:00.161599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:00.161611 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:05:00.162094 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:00.162167 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:05:00.162189 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:00.162268 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:00.162548 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 03:05:00.162795 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":449,"threads_started":1}
I20260926 03:05:00.165405 22606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:00.166563 22606 log.cc:824] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:00.167382 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:05:00.169263 22606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257: No bootstrap required, opened a new log
I20260926 03:05:00.178248 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:00.171648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44807 (local address 127.0.97.62:33009)
0926 03:05:00.171818 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:00.171824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:00.172393 (+ 569us) server_negotiation.cc:407] Connection header received
0926 03:05:00.172564 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:00.172568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:00.172626 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:00.172705 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:00.173303 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:00.174019 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:00.175802 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:00.176233 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:00.176333 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:00.177454 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:00.177459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:00.177462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:05:00.177477 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:00.177509 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:05:00.177523 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:00.177592 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:00.177839 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 03:05:00.177962 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:05:00.178645 22606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33009 } }
I20260926 03:05:00.179112 22606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:05:00.179224 22606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bde437aeb694c36ac7ec2c8bbdb8257, State: Initialized, Role: FOLLOWER
I20260926 03:05:00.179875 22606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33009 } }
I20260926 03:05:00.180083 22606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:05:00.180178 22606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:05:00.180295 22606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:00.182705 22606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33009 } }
I20260926 03:05:00.183166 22606 leader_election.cc:302] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bde437aeb694c36ac7ec2c8bbdb8257; no voters:
I20260926 03:05:00.183667 22606 leader_election.cc:288] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:05:00.183825 22616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:05:00.184533 22616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [term 1 LEADER]: Becoming Leader. State: Replica: 1bde437aeb694c36ac7ec2c8bbdb8257, State: Running, Role: LEADER
I20260926 03:05:00.185088 22616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33009 } }
I20260926 03:05:00.185771 22606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:00.187661 22617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33009 } } }
I20260926 03:05:00.187999 22617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:00.188463 22618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bde437aeb694c36ac7ec2c8bbdb8257. Latest consensus state: current_term: 1 leader_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bde437aeb694c36ac7ec2c8bbdb8257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33009 } } }
I20260926 03:05:00.188902 22618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:00.197547 22621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:05:00.201689 22621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:05:00.209726 22621 catalog_manager.cc:1347] Generated new cluster ID: b5b914224033454f89a5f4c24236e121
I20260926 03:05:00.209822 22621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:05:00.227334 22621 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:05:00.228564 22621 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:05:00.235864 22621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bde437aeb694c36ac7ec2c8bbdb8257: Generated new TSK 0
I20260926 03:05:00.236706 22621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:05:00.251812 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22539
W20260926 03:05:00.252753 22601 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37349
W20260926 03:05:00.253418 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22539: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:05:00.253569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22539
2026-09-26T03:05:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:05:00 dist-test-slave-fccw krb5kdc[22637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:05:00 dist-test-slave-fccw krb5kdc[22637](info): set up 2 sockets
Sep 26 03:05:00 dist-test-slave-fccw krb5kdc[22637](info): commencing operation
krb5kdc: starting...
W20260926 03:05:01.345350 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:05:01 dist-test-slave-fccw krb5kdc[22637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:05:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:05:01.592599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46617
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:01.913417 22650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:01.913690 22650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:01.913861 22650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:01.921427 22650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:01.921576 22650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:01.921679 22650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:01.921773 22650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:01.921855 22650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:01.931847 22650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:01.933180 22650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:01.934468 22650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:01.946151 22658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:01.946233 22656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:01.946252 22655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:01.947028 22650 server_base.cc:1034] running on GCE node
I20260926 03:05:01.947875 22650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:01.949322 22650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:01.950543 22650 hybrid_clock.cc:648] HybridClock initialized: now 1790391901950501 us; error 39 us; skew 500 ppm
I20260926 03:05:01.950891 22650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:01 dist-test-slave-fccw krb5kdc[22637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:01.960273 22650 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:05:01.961880 22650 webserver.cc:460] Webserver started at http://127.0.97.62:38621/ using document root <none> and password file <none>
I20260926 03:05:01.962563 22650 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:01.962703 22650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:01.963007 22650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:01.965974 22650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "0714abce040d40b6b76f9b2514dcf573"
format_stamp: "Formatted at 2026-09-26 03:05:01 on dist-test-slave-fccw"
server_key: "eec651183b6f8e6d406f65b8e8ed6ac5"
server_key_iv: "75d20a3c8bedfcf6cd4cdd7089c0a917"
server_key_version: "encryptionkey@0"
I20260926 03:05:01.966720 22650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "0714abce040d40b6b76f9b2514dcf573"
format_stamp: "Formatted at 2026-09-26 03:05:01 on dist-test-slave-fccw"
server_key: "eec651183b6f8e6d406f65b8e8ed6ac5"
server_key_iv: "75d20a3c8bedfcf6cd4cdd7089c0a917"
server_key_version: "encryptionkey@0"
I20260926 03:05:01.971802 22650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 03:05:01.975255 22665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:01.976570 22650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 03:05:01.976738 22650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "0714abce040d40b6b76f9b2514dcf573"
format_stamp: "Formatted at 2026-09-26 03:05:01 on dist-test-slave-fccw"
server_key: "eec651183b6f8e6d406f65b8e8ed6ac5"
server_key_iv: "75d20a3c8bedfcf6cd4cdd7089c0a917"
server_key_version: "encryptionkey@0"
I20260926 03:05:01.976949 22650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:01.982384 22650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:01.986735 22650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:01.987035 22650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:02.005275 22650 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46617
I20260926 03:05:02.005306 22716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46617 every 8 connection(s)
I20260926 03:05:02.006667 22650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:05:02.010317 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:02.016376 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22650
I20260926 03:05:02.016575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:05:02.017120 388 external_mini_cluster.cc:1428] Setting key c4ec7b321145a4476a454f92c2c740ef
I20260926 03:05:02.021143 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573: Bootstrap starting.
I20260926 03:05:02.024662 22717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:02.024946 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:02.018929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34573 (local address 127.0.97.62:46617)
0926 03:05:02.019790 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:02.019801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:02.019842 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:05:02.021630 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:02.021653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:02.021995 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:02.022311 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:02.023412 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:02.023418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:02.023432 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:05:02.023446 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:02.023505 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:05:02.023525 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:02.023683 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:02.023785 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 03:05:02.023982 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":429,"threads_started":1}
I20260926 03:05:02.025831 22717 log.cc:824] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:02.028748 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573: No bootstrap required, opened a new log
I20260926 03:05:02.028837 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:05:02.035497 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:02.032914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42343 (local address 127.0.97.62:46617)
0926 03:05:02.033100 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:02.033107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:02.033533 (+ 426us) server_negotiation.cc:407] Connection header received
0926 03:05:02.033765 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:02.033770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:02.033824 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:02.033947 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:02.034862 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:02.034867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:02.034875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:05:02.034897 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:02.034932 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:05:02.034949 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:02.035079 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:02.035165 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 03:05:02.035235 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:05:02.036923 22717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0714abce040d40b6b76f9b2514dcf573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46617 } }
I20260926 03:05:02.037333 22717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:05:02.037438 22717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0714abce040d40b6b76f9b2514dcf573, State: Initialized, Role: FOLLOWER
I20260926 03:05:02.037990 22717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0714abce040d40b6b76f9b2514dcf573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46617 } }
I20260926 03:05:02.038224 22717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:05:02.038365 22717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:05:02.038533 22717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:02.040640 22717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0714abce040d40b6b76f9b2514dcf573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46617 } }
I20260926 03:05:02.041213 22717 leader_election.cc:302] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0714abce040d40b6b76f9b2514dcf573; no voters:
I20260926 03:05:02.041757 22717 leader_election.cc:288] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:05:02.041877 22727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:05:02.042943 22727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [term 1 LEADER]: Becoming Leader. State: Replica: 0714abce040d40b6b76f9b2514dcf573, State: Running, Role: LEADER
I20260926 03:05:02.043488 22727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0714abce040d40b6b76f9b2514dcf573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46617 } }
I20260926 03:05:02.043843 22717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:02.046157 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0714abce040d40b6b76f9b2514dcf573. Latest consensus state: current_term: 1 leader_uuid: "0714abce040d40b6b76f9b2514dcf573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0714abce040d40b6b76f9b2514dcf573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46617 } } }
I20260926 03:05:02.046579 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:02.047003 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0714abce040d40b6b76f9b2514dcf573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0714abce040d40b6b76f9b2514dcf573" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46617 } } }
I20260926 03:05:02.047349 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:02.052187 22734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:05:02.057526 22734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:05:02.066201 22734 catalog_manager.cc:1347] Generated new cluster ID: 9d7291a0d1084439ba7c24282e278a46
I20260926 03:05:02.066309 22734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:05:02.095324 22734 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:05:02.095861 22734 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:05:02.104007 22734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0714abce040d40b6b76f9b2514dcf573: Generated new TSK 0
I20260926 03:05:02.104952 22734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:05:02.110651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22650
W20260926 03:05:02.112056 22712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34573
W20260926 03:05:02.112681 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 03:05:02.112810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22650
2026-09-26T03:05:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1861 ms)
[----------] 18 tests from AuthTokenIssuingTest (34193 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:05:02 dist-test-slave-fccw krb5kdc[22749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:05:02 dist-test-slave-fccw krb5kdc[22749](info): set up 2 sockets
Sep 26 03:05:02 dist-test-slave-fccw krb5kdc[22749](info): commencing operation
krb5kdc: starting...
W20260926 03:05:03.197970 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:05:03 dist-test-slave-fccw krb5kdc[22749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:05:03.449728 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45587
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45587,127.0.97.61:44971,127.0.97.60:46039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:03.767589 22762 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:05:03.767903 22762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:03.767974 22762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:03.768055 22762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:03.775122 22762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:03.775213 22762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:03.775266 22762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:03.775315 22762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:03.775374 22762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:03.784911 22762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45587,127.0.97.61:44971,127.0.97.60:46039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:03.786164 22762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:03.787427 22762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:03.798868 22767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:03.798908 22768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:03.798815 22770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:03.799800 22762 server_base.cc:1034] running on GCE node
I20260926 03:05:03.800561 22762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:03.801901 22762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:03.803105 22762 hybrid_clock.cc:648] HybridClock initialized: now 1790391903803024 us; error 55 us; skew 500 ppm
I20260926 03:05:03.803416 22762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:03 dist-test-slave-fccw krb5kdc[22749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:03.806432 22762 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:05:03.807725 22762 webserver.cc:460] Webserver started at http://127.0.97.62:33503/ using document root <none> and password file <none>
I20260926 03:05:03.808337 22762 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:03.808427 22762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:03.808684 22762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:03.811455 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0"
format_stamp: "Formatted at 2026-09-26 03:05:03 on dist-test-slave-fccw"
server_key: "256858beb4e442677a539be50b007b9a"
server_key_iv: "11cf11ae0f911611d30726b2b2b94525"
server_key_version: "encryptionkey@0"
I20260926 03:05:03.812173 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0"
format_stamp: "Formatted at 2026-09-26 03:05:03 on dist-test-slave-fccw"
server_key: "256858beb4e442677a539be50b007b9a"
server_key_iv: "11cf11ae0f911611d30726b2b2b94525"
server_key_version: "encryptionkey@0"
I20260926 03:05:03.817059 22762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 03:05:03.820195 22777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:03.821519 22762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:05:03.821692 22762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0"
format_stamp: "Formatted at 2026-09-26 03:05:03 on dist-test-slave-fccw"
server_key: "256858beb4e442677a539be50b007b9a"
server_key_iv: "11cf11ae0f911611d30726b2b2b94525"
server_key_version: "encryptionkey@0"
I20260926 03:05:03.821900 22762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:03.845870 22762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:03.849721 22762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:03.849992 22762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:03.868343 22762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45587
I20260926 03:05:03.868355 22828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45587 every 8 connection(s)
I20260926 03:05:03.869693 22762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:05:03.870558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22762
I20260926 03:05:03.870784 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:05:03.871585 388 external_mini_cluster.cc:1428] Setting key 0f4272949ece684d5079b1cf212a51b0
I20260926 03:05:03.874305 22829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:03.883466 22829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 26 03:05:03 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:03.907939 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:03.886512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45587 (local address 127.0.0.1:43596)
0926 03:05:03.887626 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:03.887663 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:03.889386 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:03.890149 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:03.890155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:03.890884 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:03.891190 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:03.891206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:03.892725 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:03.892733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:03.893981 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:03.894010 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:03.894925 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:03.894931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:03.895103 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:03.895777 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:03.895797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:03.900423 (+ 4626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:03.903415 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:03.903428 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:03.903441 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:03.904062 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:03.904460 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:03.904468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:03.904474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:03.904685 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:03.905475 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:03.905485 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:03.905740 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 03:05:03.906023 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 03:05:03.906591 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":96,"thread_start_us":697,"threads_started":1}
I20260926 03:05:03.907963 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:03.886511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.0.97.62:45587)
0926 03:05:03.887360 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:03.887376 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:03.887771 (+ 395us) server_negotiation.cc:407] Connection header received
0926 03:05:03.889737 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:03.889742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:03.889953 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:03.890138 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:03.891396 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:03.892579 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:03.894156 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:03.894700 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:03.894899 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:03.900652 (+ 5753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:03.900660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:03.900670 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:03.900699 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:03.903295 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:03.904202 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:03.904209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:03.904211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:03.904348 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:03.904797 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:03.904800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:03.904802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:03.905331 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:03.905498 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:03.906141 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 03:05:03.906464 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":572,"thread_start_us":369,"threads_started":1}
I20260926 03:05:03.910835 22829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:03.912071 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:03.911496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34218)
0926 03:05:03.911762 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:03.911901 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260926 03:05:03.912315 22780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
I20260926 03:05:03.913947 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:03.913497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34220)
0926 03:05:03.913670 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:03.913774 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 03:05:03.914136 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:05:03.929539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44971
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45587,127.0.97.61:44971,127.0.97.60:46039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:05:03.963495 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } attempt: 1
I20260926 03:05:03.964726 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:03.964206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34228)
0926 03:05:03.964433 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:03.964541 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 03:05:03.965843 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:03.965435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34238)
0926 03:05:03.965586 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:03.965667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:05:03.966080 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
I20260926 03:05:04.034350 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } attempt: 2
I20260926 03:05:04.035707 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.035109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34244)
0926 03:05:04.035371 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.035484 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:05:04.036895 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.036379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34248)
0926 03:05:04.036559 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.036655 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:05:04.037225 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
I20260926 03:05:04.128511 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } attempt: 3
I20260926 03:05:04.129740 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.129233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34258)
0926 03:05:04.129457 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.129569 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:05:04.130795 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.130385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34268)
0926 03:05:04.130525 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.130608 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 03:05:04.130978 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
W20260926 03:05:04.256345 22836 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:05:04.256640 22836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:04.256704 22836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:04.256784 22836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:04.263813 22836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:04.263901 22836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:04.263957 22836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:04.263998 22836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:04.264051 22836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:04.273509 22836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45587,127.0.97.61:44971,127.0.97.60:46039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:04.274227 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } attempt: 4
I20260926 03:05:04.274860 22836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:04.275465 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.274886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34278)
0926 03:05:04.275263 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.275342 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260926 03:05:04.276413 22836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:05:04.276546 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.276143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34286)
0926 03:05:04.276300 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.276404 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:05:04.276759 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44971: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44971: connect: Connection refused (error 111)
W20260926 03:05:04.287869 22841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:04.287796 22844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:04.287834 22842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:04.288938 22836 server_base.cc:1034] running on GCE node
I20260926 03:05:04.289721 22836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:04.291011 22836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:04.292217 22836 hybrid_clock.cc:648] HybridClock initialized: now 1790391904292159 us; error 29 us; skew 500 ppm
I20260926 03:05:04.292534 22836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:04.295672 22836 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:05:04.297099 22836 webserver.cc:460] Webserver started at http://127.0.97.61:34277/ using document root <none> and password file <none>
I20260926 03:05:04.297766 22836 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:04.297852 22836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:04.298115 22836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:04.300879 22836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data/instance:
uuid: "a664a0b4e137406499287b75462d381f"
format_stamp: "Formatted at 2026-09-26 03:05:04 on dist-test-slave-fccw"
server_key: "d4a26f7e9232d4e5ee89741342c644e1"
server_key_iv: "9934702632da1be523f6ddcf0ed84591"
server_key_version: "encryptionkey@0"
I20260926 03:05:04.301659 22836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/wal/instance:
uuid: "a664a0b4e137406499287b75462d381f"
format_stamp: "Formatted at 2026-09-26 03:05:04 on dist-test-slave-fccw"
server_key: "d4a26f7e9232d4e5ee89741342c644e1"
server_key_iv: "9934702632da1be523f6ddcf0ed84591"
server_key_version: "encryptionkey@0"
I20260926 03:05:04.306406 22836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 03:05:04.309571 22851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:04.310782 22836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:05:04.310945 22836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/wal
uuid: "a664a0b4e137406499287b75462d381f"
format_stamp: "Formatted at 2026-09-26 03:05:04 on dist-test-slave-fccw"
server_key: "d4a26f7e9232d4e5ee89741342c644e1"
server_key_iv: "9934702632da1be523f6ddcf0ed84591"
server_key_version: "encryptionkey@0"
I20260926 03:05:04.311141 22836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:04.324103 22836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:04.327977 22836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:04.328227 22836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:04.345938 22836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44971
I20260926 03:05:04.345960 22902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44971 every 8 connection(s)
I20260926 03:05:04.347306 22836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:05:04.350862 22903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:04.351537 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22836
I20260926 03:05:04.351719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-1/wal/instance
I20260926 03:05:04.352309 388 external_mini_cluster.cc:1428] Setting key fe884554b818fecfc4a35e3968ec6ecb
I20260926 03:05:04.360841 22903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260926 03:05:04.383008 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.364131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.0.97.62:45587)
0926 03:05:04.364381 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.364389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.365596 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 03:05:04.367495 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.367503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.367592 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.367711 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.369226 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.370356 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.371879 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.372625 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.372760 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.376576 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.376583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.376590 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.376633 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.379928 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.380609 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.380613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.380616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.380751 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.381191 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.381197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.381202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.381601 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.381731 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.382601 (+ 870us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.382765 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:05:04.384634 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.364570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45587 (local address 127.0.0.1:43612)
0926 03:05:04.365425 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.365474 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:04.367213 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:04.367894 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:04.367902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:04.368413 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:04.369046 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.369072 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.370525 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.370528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.371726 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.371738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.372832 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.372837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.373095 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.373798 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:04.373819 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:04.376321 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:04.380052 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.380058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.380071 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.380483 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.380842 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.380845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.380848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.381077 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.381741 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:04.381749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:04.382216 (+ 467us) client_negotiation.cc:769] Sending connection context
0926 03:05:04.382524 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 03:05:04.383104 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":397,"threads_started":1}
I20260926 03:05:04.386384 22903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:05:04.404232 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.387383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.0.97.61:44971)
0926 03:05:04.388030 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.388036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.388059 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:05:04.388099 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.388105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.388269 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.388384 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.389355 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.390417 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.391692 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.392651 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.392787 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.396610 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.396628 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.396650 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.396694 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.400946 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.401845 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.401851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.401854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.402003 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.402360 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.402364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.402368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.402881 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.403108 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.403562 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.403893 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":495,"thread_start_us":368,"threads_started":1}
I20260926 03:05:04.404255 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.387156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34298)
0926 03:05:04.387391 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.387411 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:04.387536 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:04.388429 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:04.388436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:04.388930 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:04.389178 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.389193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.390598 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.390605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.391456 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.391466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.392792 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.392796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.392949 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.393586 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:04.393613 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:04.396360 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:04.401143 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.401151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.401157 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.401720 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.402110 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.402117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.402123 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.402249 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.403041 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:04.403049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:04.403251 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 03:05:04.403406 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 03:05:04.403726 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:05:04.406806 22903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:04.407855 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.407458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54328)
0926 03:05:04.407602 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.407708 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:05:04.408082 22855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
I20260926 03:05:04.409595 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.409295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54332)
0926 03:05:04.409427 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.409505 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 03:05:04.409817 22903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:05:04.416481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46039
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45587,127.0.97.61:44971,127.0.97.60:46039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:05:04.459111 22903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 1
I20260926 03:05:04.460407 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.459827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54344)
0926 03:05:04.460074 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.460197 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 03:05:04.461606 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.461141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54354)
0926 03:05:04.461325 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.461459 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:05:04.461879 22903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
I20260926 03:05:04.530076 22903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 2
I20260926 03:05:04.531245 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.530710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54356)
0926 03:05:04.530932 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.531052 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 03:05:04.532364 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.531984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54362)
0926 03:05:04.532131 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.532224 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:05:04.532634 22903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
I20260926 03:05:04.576051 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } attempt: 5
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:05:04.589406 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.576851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34308)
0926 03:05:04.577074 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.577095 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:04.577225 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:04.577603 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:04.577610 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:04.578043 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:04.578224 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.578233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.579227 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.579238 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.580145 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.580157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.580871 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.580877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.581123 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.581781 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:04.581797 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:04.583996 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:04.587013 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.587019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.587023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.587394 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.587752 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.587756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.587760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.587897 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.588405 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:04.588409 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:04.588567 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 03:05:04.588710 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 03:05:04.589032 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 03:05:04.589404 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.576947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.0.97.61:44971)
0926 03:05:04.577187 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.577194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.577213 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:05:04.577391 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.577395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.577462 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.577564 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.578372 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.579072 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.580283 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.580722 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.580844 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.584193 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.584198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.584201 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.584234 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.586877 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.587514 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.587521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.587524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.587642 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.588007 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.588014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.588018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.588288 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.588437 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.588828 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.589081 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 03:05:04.590512 22829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:04.591521 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.591104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54366)
0926 03:05:04.591302 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.591384 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 03:05:04.592569 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.592257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54382)
0926 03:05:04.592402 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.592482 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 03:05:04.592810 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
I20260926 03:05:04.623860 22903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 3
I20260926 03:05:04.625120 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.624554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54398)
0926 03:05:04.624773 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.624903 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 03:05:04.626243 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.625808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54402)
0926 03:05:04.626026 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.626112 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260926 03:05:04.626471 22903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
I20260926 03:05:04.644055 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 1
I20260926 03:05:04.645171 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.644697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54412)
0926 03:05:04.644878 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.644964 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 03:05:04.646039 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.645734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54428)
0926 03:05:04.645866 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.645938 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:05:04.646263 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
I20260926 03:05:04.714488 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 2
I20260926 03:05:04.715747 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.715185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54432)
0926 03:05:04.715428 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.715540 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 03:05:04.716831 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.716414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54442)
0926 03:05:04.716605 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.716701 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 03:05:04.717075 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
W20260926 03:05:04.741657 22910 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:05:04.741971 22910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:04.742038 22910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:04.742120 22910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:04.749285 22910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:04.749372 22910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:04.749430 22910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:04.749481 22910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:04.749540 22910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:04.759188 22910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45587,127.0.97.61:44971,127.0.97.60:46039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:04.760401 22910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:04.761730 22910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:05:04.769667 22903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 4
I20260926 03:05:04.770720 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.770266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54448)
0926 03:05:04.770495 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.770574 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 03:05:04.771973 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.771525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54458)
0926 03:05:04.771723 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.771817 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:05:04.772279 22903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
W20260926 03:05:04.773255 22916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:04.773430 22915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:04.773319 22918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:04.774376 22910 server_base.cc:1034] running on GCE node
I20260926 03:05:04.775112 22910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:04.776429 22910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:04.777659 22910 hybrid_clock.cc:648] HybridClock initialized: now 1790391904777591 us; error 43 us; skew 500 ppm
I20260926 03:05:04.777971 22910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:04.781067 22910 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:05:04.782419 22910 webserver.cc:460] Webserver started at http://127.0.97.60:46065/ using document root <none> and password file <none>
I20260926 03:05:04.783102 22910 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:04.783227 22910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:04.783545 22910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:04.786471 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data/instance:
uuid: "25bb6db684b044deba4097e915358a75"
format_stamp: "Formatted at 2026-09-26 03:05:04 on dist-test-slave-fccw"
server_key: "28862a8d3ef626646b1ab4aaea382a5e"
server_key_iv: "ae60b022fe765c01d5935960b2c3db32"
server_key_version: "encryptionkey@0"
I20260926 03:05:04.787245 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/wal/instance:
uuid: "25bb6db684b044deba4097e915358a75"
format_stamp: "Formatted at 2026-09-26 03:05:04 on dist-test-slave-fccw"
server_key: "28862a8d3ef626646b1ab4aaea382a5e"
server_key_iv: "ae60b022fe765c01d5935960b2c3db32"
server_key_version: "encryptionkey@0"
I20260926 03:05:04.792662 22910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 03:05:04.796171 22925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:04.797503 22910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 03:05:04.797674 22910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/wal
uuid: "25bb6db684b044deba4097e915358a75"
format_stamp: "Formatted at 2026-09-26 03:05:04 on dist-test-slave-fccw"
server_key: "28862a8d3ef626646b1ab4aaea382a5e"
server_key_iv: "ae60b022fe765c01d5935960b2c3db32"
server_key_version: "encryptionkey@0"
I20260926 03:05:04.797873 22910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:04.818225 22910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:04.822630 22910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:04.822899 22910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:04.823326 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 3
I20260926 03:05:04.824373 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.823865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54462)
0926 03:05:04.824091 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.824177 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 03:05:04.825605 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.825145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54474)
0926 03:05:04.825359 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.825459 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 03:05:04.825863 22829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46039: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46039: connect: Connection refused (error 111)
I20260926 03:05:04.842267 22910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46039
I20260926 03:05:04.842294 22976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46039 every 8 connection(s)
I20260926 03:05:04.843608 22910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:05:04.847415 22977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:04.850160 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 22910
I20260926 03:05:04.850329 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790391557890803-388-0/minicluster-data/master-2/wal/instance
I20260926 03:05:04.850899 388 external_mini_cluster.cc:1428] Setting key 02ac00a714dc0c4e41309e80c0120074
I20260926 03:05:04.855450 22977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:04.873467 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.858465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.97.62:45587)
0926 03:05:04.858633 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.858638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.859414 (+ 776us) server_negotiation.cc:407] Connection header received
0926 03:05:04.859615 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.859620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.859691 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.859802 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.860850 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.861610 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.862914 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.863357 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.863487 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.867304 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.867309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.867312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.867343 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.870470 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.871170 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.871178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.871181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.871340 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.871807 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.871811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.871814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.872204 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.872323 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.872842 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.873069 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:05:04.873934 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.858507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45587 (local address 127.0.0.1:43624)
0926 03:05:04.859240 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.859305 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:04.859489 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:04.859861 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:04.859868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:04.860467 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:04.860699 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.860708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.861761 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.861771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.862769 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.862782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.863574 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.863582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.863738 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.864922 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:04.864949 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:04.867096 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:04.870610 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.870619 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.870624 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.871016 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.871483 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.871488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.871491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.871671 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.872409 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:04.872414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:04.872551 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 03:05:04.872721 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 03:05:04.873066 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":407,"threads_started":1}
I20260926 03:05:04.873960 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.852738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.0.97.60:46039)
0926 03:05:04.853570 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.853585 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.853626 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:05:04.855207 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.855229 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.855587 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.855920 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.857777 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.859745 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.860987 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.861621 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.861811 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.865705 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.865715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.865727 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.865764 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.869088 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.869917 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.869925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.869934 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.870118 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.870524 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.870529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.870532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.871147 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.871427 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.871876 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.872491 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":399,"threads_started":1}
I20260926 03:05:04.875554 22977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:04.888840 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.876359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44971 (local address 127.0.0.1:34322)
0926 03:05:04.876507 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.876526 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:04.876626 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:04.877438 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:04.877443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:04.877921 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:04.878061 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.878068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.879027 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.879032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.879894 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.879904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.880587 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.880592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.880768 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.881341 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:04.881357 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:04.883480 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:04.886555 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.886562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.886566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.886914 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.887273 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.887277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.887281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.887427 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.887908 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:04.887913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:04.888061 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 03:05:04.888192 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 03:05:04.888458 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 03:05:04.888852 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.876572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.0.97.61:44971)
0926 03:05:04.876766 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.876779 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.876793 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:05:04.876839 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.876842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.876896 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.876993 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.878194 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.878882 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.880024 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.880431 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.880569 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.883650 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.883656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.883661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.883700 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.886433 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.887022 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.887029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.887033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.887156 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.887530 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.887536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.887545 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.887798 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.887941 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.888285 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.888489 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:05:04.889803 22977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:04.890911 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.877301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.0.97.62:45587)
0926 03:05:04.877453 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.877458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.877474 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:05:04.877634 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.877642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.877708 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.877812 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.879061 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.879800 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.880984 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.881406 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.881529 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.885507 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.885512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.885515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.885552 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.888081 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.888716 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.888722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.888725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.888848 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.889352 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.889357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.889360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.889642 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.889928 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.890193 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.890401 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 26 03:05:04 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:05:04.903117 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.890439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54500)
0926 03:05:04.890656 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:04.890675 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:04.890780 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:04.891186 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:04.891189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:04.891585 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:04.891754 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.891766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.892927 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.892932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.893898 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:04.893909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.894778 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.894783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:04.894992 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.895559 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:04.895576 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:04.897445 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:04.900600 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.900604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.900608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.901130 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.901477 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:04.901481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:04.901485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:04.901671 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:04.902287 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:04.902293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:04.902456 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 03:05:04.902593 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 03:05:04.902849 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 03:05:04.903542 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.890463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.0.97.60:46039)
0926 03:05:04.890618 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.890629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.890741 (+ 112us) server_negotiation.cc:407] Connection header received
0926 03:05:04.890871 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.890875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.890954 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.891049 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.891917 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.892799 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.894045 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.894644 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.894738 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.897612 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.897616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.897620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.897645 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.900450 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.901234 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.901238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.901240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.901343 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.901761 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.901765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.901766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.902036 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.902305 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.902807 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.903040 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 03:05:04.907590 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:04.892601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.0.97.61:44971)
0926 03:05:04.892742 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:04.892746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:04.892760 (+ 14us) server_negotiation.cc:407] Connection header received
0926 03:05:04.892896 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:04.892900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:04.892970 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:04.893090 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:04.894211 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.895239 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.896468 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:04.896882 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:04.897052 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:04.901687 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:04.901693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:04.901701 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:04.901743 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:04.904953 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.905855 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.905859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.905861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.905975 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:04.906402 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:04.906405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:04.906407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:04.906642 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:04.906832 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:04.907158 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 03:05:04.907338 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:05:04.909317 22977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: Bootstrap starting.
I20260926 03:05:04.912055 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:05:04.912757 22977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:04.913903 22977 log.cc:824] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:04.914652 22804 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43634
I20260926 03:05:04.919858 22977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: No bootstrap required, opened a new log
I20260926 03:05:04.928126 22977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:04.928467 22977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bb6db684b044deba4097e915358a75, State: Initialized, Role: FOLLOWER
I20260926 03:05:04.928916 22977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:04.929934 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:04.930341 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:04.930984 22977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 03:05:04.939683 22994 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:05:04.939822 22994 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:05:04.938194 22910 master_runner.cc:186] Error getting master registration for 127.0.97.62:45587: OK, instance_id { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" instance_seqno: 1790391903863343 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:05:04.949054 22910 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:05:05.003087 22829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 4
Sep 26 03:05:05 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:05:05.017619 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:05.003842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54510)
0926 03:05:05.004071 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:05.004091 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:05.004229 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:05.004644 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:05.004649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:05.005267 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:05.005500 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:05.005514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.006545 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.006553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:05.007557 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:05.007569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.008329 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.008334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:05.008561 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:05.009201 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:05.009219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:05.011518 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:05.014893 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:05.014899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:05.014903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:05.015386 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:05.015782 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:05.015788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:05.015792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:05.015960 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:05.016588 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:05.016593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:05.016755 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 03:05:05.016913 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 03:05:05.017202 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 03:05:05.017671 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:05.004134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.0.97.60:46039)
0926 03:05:05.004349 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.004356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.004373 (+ 17us) server_negotiation.cc:407] Connection header received
0926 03:05:05.004436 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.004440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.004500 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.004616 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.005661 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.006394 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.007699 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.008176 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.008313 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:05.011690 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:05.011694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:05.011698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:05.011728 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:05.014770 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.015506 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.015514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.015520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.015684 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.016069 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.016073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.016076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.016472 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:05.016612 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:05.017051 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 03:05:05.017257 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 03:05:05.023671 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0: Bootstrap starting.
I20260926 03:05:05.027864 22829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:05.028908 22829 log.cc:824] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:05.031895 22829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0: No bootstrap required, opened a new log
I20260926 03:05:05.039184 22829 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:05.039572 22829 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d5c6d4bffc42df886b5d340eaaa8e0, State: Initialized, Role: FOLLOWER
I20260926 03:05:05.040067 22829 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:05.041028 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.041431 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:05.042141 22829 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:05.042272 22804 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:05:05.042498 22804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:05.046125 22804 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:05.048125 22804 leader_election.cc:288] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [CANDIDATE]: Term 1 election: Requested vote from peers a664a0b4e137406499287b75462d381f (127.0.97.61:44971), 25bb6db684b044deba4097e915358a75 (127.0.97.60:46039)
I20260926 03:05:05.049023 22952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25bb6db684b044deba4097e915358a75"
I20260926 03:05:05.049645 22952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:05.053946 22952 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d5c6d4bffc42df886b5d340eaaa8e0 in term 1.
I20260926 03:05:05.054605 22781 leader_election.cc:302] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bb6db684b044deba4097e915358a75, c6d5c6d4bffc42df886b5d340eaaa8e0; no voters:
I20260926 03:05:05.055068 22997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:05:05.061338 22997 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [term 1 LEADER]: Becoming Leader. State: Replica: c6d5c6d4bffc42df886b5d340eaaa8e0, State: Running, Role: LEADER
I20260926 03:05:05.062583 22997 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:05.068373 23011 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d5c6d4bffc42df886b5d340eaaa8e0. Latest consensus state: current_term: 1 leader_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.071161 23011 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:05.071512 22903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } attempt: 5
W20260926 03:05:05.075106 22931 tablet.cc:2369] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:05:05 dist-test-slave-fccw krb5kdc[22749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260926 03:05:05.083056 23008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:05:05.083173 23008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:05:05.069868 22762 master_runner.cc:186] Error getting master registration for 127.0.97.61:44971: OK, instance_id { permanent_uuid: "a664a0b4e137406499287b75462d381f" instance_seqno: 1790391904340912 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:05:05.084692 22762 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:05:05.093041 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:05.072516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.0.97.60:46039)
0926 03:05:05.072696 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.072702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.072722 (+ 20us) server_negotiation.cc:407] Connection header received
0926 03:05:05.072889 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.072894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.072970 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.073092 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.074196 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.075252 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.077101 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.077794 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.077920 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:05.082124 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:05.082130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:05.082134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:05.082170 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:05.085911 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.086780 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.086790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.086796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.087070 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.087829 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.087834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.087837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.088715 (+ 878us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:05.089791 (+ 1076us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:05.090329 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 03:05:05.091374 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 03:05:05.093097 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:05.072151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46039 (local address 127.0.0.1:54512)
0926 03:05:05.072590 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:05.072614 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:05.072747 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:05.073284 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:05.073289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:05.073799 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:05.074004 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:05.074016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.075414 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.075422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:05.076898 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:05.076912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.078141 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.078147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:05.078352 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:05.079044 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:05.079068 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:05.081895 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:05.086103 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:05.086110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:05.086114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:05.086632 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:05.087313 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:05.087336 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:05.087339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:05.087659 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:05.088932 (+ 1273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:05.088940 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:05.089311 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 03:05:05.089747 (+ 436us) client_negotiation.cc:240] Negotiation successful
0926 03:05:05.091182 (+ 1435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275}
W20260926 03:05:05.102550 22783 tablet.cc:2369] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:05:05.106004 22903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f: Bootstrap starting.
I20260926 03:05:05.110829 22903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:05.112362 22903 log.cc:824] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:05.115016 23015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:05:05.118157 23015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:05:05.124615 22903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f: No bootstrap required, opened a new log
I20260926 03:05:05.135864 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:05.096868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.97.62:45587)
0926 03:05:05.097181 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.097190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.115445 (+ 18255us) server_negotiation.cc:407] Connection header received
0926 03:05:05.115834 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.115841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.115931 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.116057 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.118897 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.120026 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.122133 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.122775 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.122866 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:05.126485 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:05.126492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:05.126496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:05.126540 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:05.130326 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.131250 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.131256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.131260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.131420 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.132473 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.132481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.132485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.132854 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:05.134298 (+ 1444us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:05.135207 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 03:05:05.135443 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260926 03:05:05.138286 22952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [term 1 FOLLOWER]: Refusing update from remote peer c6d5c6d4bffc42df886b5d340eaaa8e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:05:05.139261 23011 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:05:05.140540 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:05.117081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.0.97.60:46039)
0926 03:05:05.117278 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.117286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.117877 (+ 591us) server_negotiation.cc:407] Connection header received
0926 03:05:05.118266 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.118271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.118360 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.118429 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.122163 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.123209 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.126652 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.127282 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.127383 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:05.132235 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:05.132242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:05.132247 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:05.132294 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:05.135822 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.137513 (+ 1691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.137518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.137521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.137664 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.138328 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.138334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.138337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.138662 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:05.138895 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:05.139958 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0926 03:05:05.140200 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 03:05:05.145052 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:05.096435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.0.97.61:44971)
0926 03:05:05.096602 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.096609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.116376 (+ 19767us) server_negotiation.cc:407] Connection header received
0926 03:05:05.117347 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.117353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.117432 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.117501 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.119885 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.120940 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.126620 (+ 5680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:05.127411 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:05.127520 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:05.137416 (+ 9896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:05.137424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:05.137429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:05.137467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:05.141168 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.142323 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.142329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.142332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.142506 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:05.142915 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:05.142922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:05.142927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:05.143321 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:05.143559 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:05.144399 (+ 840us) server_negotiation.cc:299] Negotiation successful
0926 03:05:05.144640 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 03:05:05.154210 22903 raft_consensus.cc:348] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:05.154608 22903 raft_consensus.cc:729] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a664a0b4e137406499287b75462d381f, State: Initialized, Role: FOLLOWER
I20260926 03:05:05.155172 22903 consensus_queue.cc:260] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } }
I20260926 03:05:05.155319 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d5c6d4bffc42df886b5d340eaaa8e0. Latest consensus state: current_term: 1 leader_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.155619 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:05.165460 23011 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.165712 23011 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:05.166165 23022 mvcc.cc:205] Tried to move back new op lower bound from 7333445243433975808 to 7333445243148009472. Current Snapshot: MvccSnapshot[applied={T|T < 7333445243433975808}]
I20260926 03:05:05.168867 22983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.169128 22983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:05.168637 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.169694 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:05.171082 22903 sys_catalog.cc:564] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:05.172897 23015 catalog_manager.cc:1347] Generated new cluster ID: 01c949270f0a498e9d23ae337a392723
I20260926 03:05:05.173008 23015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:05:05.173749 22877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:05.177143 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a664a0b4e137406499287b75462d381f"
I20260926 03:05:05.178550 22877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [term 1 FOLLOWER]: Refusing update from remote peer c6d5c6d4bffc42df886b5d340eaaa8e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:05:05.181739 22878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d5c6d4bffc42df886b5d340eaaa8e0 in term 1.
I20260926 03:05:05.183799 22997 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:05:05.230643 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d5c6d4bffc42df886b5d340eaaa8e0. Latest consensus state: current_term: 1 leader_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.230993 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:05.231293 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.231770 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:05.239238 23041 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:05:05.243089 22997 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6d5c6d4bffc42df886b5d340eaaa8e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45587 } } peers { permanent_uuid: "a664a0b4e137406499287b75462d381f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44971 } } peers { permanent_uuid: "25bb6db684b044deba4097e915358a75" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46039 } } }
I20260926 03:05:05.243462 22997 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:05.260013 23015 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:05:05.261368 23015 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:05:05.275725 23015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d5c6d4bffc42df886b5d340eaaa8e0: Generated new TSK 0
I20260926 03:05:05.276774 23015 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:05:05.345041 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.338869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.0.97.61:44971)
0926 03:05:05.339071 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.339078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.342705 (+ 3627us) server_negotiation.cc:407] Connection header received
0926 03:05:05.343005 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.343015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.343144 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.343278 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.344744 (+ 1466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34352: BlockingRecv error: recv got EOF from 127.0.0.1:34352 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 03:05:05.347069 22832 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.338906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.97.62:45587)
0926 03:05:05.339153 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.339162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.343657 (+ 4495us) server_negotiation.cc:407] Connection header received
0926 03:05:05.345110 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.345116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.345198 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.345289 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.346841 (+ 1552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43666: BlockingRecv error: recv got EOF from 127.0.0.1:43666 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 03:05:05.347086 22980 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.344368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.0.97.60:46039)
0926 03:05:05.344555 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.344561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.345651 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 03:05:05.345902 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.345911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.346023 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.346102 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.346843 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54522: BlockingRecv error: recv got EOF from 127.0.0.1:54522 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 03:05:05.381878 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.379300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.0.97.61:44971)
0926 03:05:05.379491 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.379497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.379970 (+ 473us) server_negotiation.cc:407] Connection header received
0926 03:05:05.380278 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.380284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.380445 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.380526 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.381605 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34358: BlockingRecv error: recv got EOF from 127.0.0.1:34358 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 03:05:05.392877 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.390750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.0.97.61:44971)
0926 03:05:05.390922 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.390928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.391404 (+ 476us) server_negotiation.cc:407] Connection header received
0926 03:05:05.391657 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.391665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.391751 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.391819 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.392679 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34360: BlockingRecv error: recv got EOF from 127.0.0.1:34360 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 03:05:05.405776 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.401935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.0.97.61:44971)
0926 03:05:05.402086 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.402091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.404112 (+ 2021us) server_negotiation.cc:407] Connection header received
0926 03:05:05.404371 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.404377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.404491 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.404568 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.405538 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34364: BlockingRecv error: recv got EOF from 127.0.0.1:34364 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 03:05:05.417224 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.414854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.0.97.61:44971)
0926 03:05:05.415056 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.415062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.415575 (+ 513us) server_negotiation.cc:407] Connection header received
0926 03:05:05.415903 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.415909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.416011 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.416080 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.416972 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34378: BlockingRecv error: recv got EOF from 127.0.0.1:34378 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260926 03:05:05.431911 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.429482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.0.97.61:44971)
0926 03:05:05.429665 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.429671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.430160 (+ 489us) server_negotiation.cc:407] Connection header received
0926 03:05:05.430451 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.430458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.430547 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.430619 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.431659 (+ 1040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34386: BlockingRecv error: recv got EOF from 127.0.0.1:34386 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 03:05:05.454782 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.452328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.0.97.61:44971)
0926 03:05:05.452515 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.452521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.453051 (+ 530us) server_negotiation.cc:407] Connection header received
0926 03:05:05.453366 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.453373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.453475 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.453558 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.454537 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34390: BlockingRecv error: recv got EOF from 127.0.0.1:34390 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 03:05:05.493435 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.491346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.97.61:44971)
0926 03:05:05.491550 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.491557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.491953 (+ 396us) server_negotiation.cc:407] Connection header received
0926 03:05:05.492142 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.492148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.492242 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.492429 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.493149 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34398: BlockingRecv error: recv got EOF from 127.0.0.1:34398 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 03:05:05.563109 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.561157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.0.97.61:44971)
0926 03:05:05.561392 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.561399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.561780 (+ 381us) server_negotiation.cc:407] Connection header received
0926 03:05:05.561944 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.561950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.562049 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.562179 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.562884 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34402: BlockingRecv error: recv got EOF from 127.0.0.1:34402 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 03:05:05.699232 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.697366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.0.97.61:44971)
0926 03:05:05.697605 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.697611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.697901 (+ 290us) server_negotiation.cc:407] Connection header received
0926 03:05:05.698093 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.698097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.698168 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.698253 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.698982 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34416: BlockingRecv error: recv got EOF from 127.0.0.1:34416 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:05:05.941780 22994 catalog_manager.cc:1259] Loaded cluster ID: 01c949270f0a498e9d23ae337a392723
I20260926 03:05:05.941896 22994 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: loading cluster ID for follower catalog manager: success
I20260926 03:05:05.944654 22994 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: acquiring CA information for follower catalog manager: success
I20260926 03:05:05.946540 22994 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25bb6db684b044deba4097e915358a75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:05:05.968569 22907 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:05.966764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.0.97.61:44971)
0926 03:05:05.967025 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:05.967031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:05.967174 (+ 143us) server_negotiation.cc:407] Connection header received
0926 03:05:05.967419 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:05.967423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:05.967493 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:05.967600 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:05.968349 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34432: BlockingRecv error: recv got EOF from 127.0.0.1:34432 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 03:05:06.245049 23041 catalog_manager.cc:1259] Loaded cluster ID: 01c949270f0a498e9d23ae337a392723
I20260926 03:05:06.245186 23041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f: loading cluster ID for follower catalog manager: success
I20260926 03:05:06.248956 23041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f: acquiring CA information for follower catalog manager: success
I20260926 03:05:06.251417 23041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a664a0b4e137406499287b75462d381f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:05:06.500357 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:06.490699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.97.61:44971)
0926 03:05:06.493576 (+ 2877us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:06.493596 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:06.493910 (+ 314us) server_negotiation.cc:407] Connection header received
0926 03:05:06.494116 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:06.494120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:06.494267 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:06.494395 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:06.494929 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.495737 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.497776 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.498357 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.498455 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:06.499062 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:06.499623 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:06.499766 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:06.499875 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 03:05:06.499937 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2713,"thread_start_us":2507,"threads_started":1}
I20260926 03:05:06.513496 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:06.507201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.0.97.60:46039)
0926 03:05:06.507836 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:06.507841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:06.507856 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:05:06.507942 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:06.507945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:06.508081 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:06.508214 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:06.508662 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.509399 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.511231 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.511621 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.511711 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:06.512262 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:06.512778 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:06.512891 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:06.513093 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 03:05:06.513161 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":430,"threads_started":1}
I20260926 03:05:06.538164 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:06.520928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.0.97.62:45587)
0926 03:05:06.529855 (+ 8927us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:06.529865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:06.529892 (+ 27us) server_negotiation.cc:407] Connection header received
0926 03:05:06.530048 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:06.530057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:06.530309 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:06.530538 (+ 229us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:06.531203 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.532376 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.535001 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.535654 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.535782 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:06.536490 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:06.536943 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:06.537092 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:06.537748 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 03:05:06.537830 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8764,"thread_start_us":8630,"threads_started":1}
I20260926 03:05:06.540329 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:06.521228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.97.61:44971)
0926 03:05:06.521421 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:06.521429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:06.531335 (+ 9906us) server_negotiation.cc:407] Connection header received
0926 03:05:06.531674 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:06.531680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:06.531764 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:06.531835 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:06.532777 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.534335 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.537325 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.538284 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.538407 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:06.539219 (+ 812us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:06.539381 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:06.539459 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:06.539772 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:05:06.539883 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 03:05:06.545065 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:06.531563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.0.97.60:46039)
0926 03:05:06.531738 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:06.531744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:06.537666 (+ 5922us) server_negotiation.cc:407] Connection header received
0926 03:05:06.538928 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:06.538934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:06.539033 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:06.539099 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:06.539677 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.540776 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.542841 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:06.543483 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:06.543633 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:06.544264 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:06.544406 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:06.544500 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:06.544684 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 03:05:06.544760 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:05:06.549795 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22762
I20260926 03:05:06.550917 22824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:06.662626 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22762
I20260926 03:05:06.683419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22836
I20260926 03:05:06.688858 22898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:06.794271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22836
I20260926 03:05:06.813781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 22910
I20260926 03:05:06.814805 22972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:06.923291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 22910
2026-09-26T03:05:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4816 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:05:07 dist-test-slave-fccw krb5kdc[23129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:05:07 dist-test-slave-fccw krb5kdc[23129](info): set up 2 sockets
Sep 26 03:05:07 dist-test-slave-fccw krb5kdc[23129](info): commencing operation
krb5kdc: starting...
W20260926 03:05:08.033576 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/test-user.keytab.
Sep 26 03:05:08 dist-test-slave-fccw krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:05:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:05:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260926 03:05:08.283670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45205
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45205,127.0.97.61:36251,127.0.97.60:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:05:08.604280 23142 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:05:08.604663 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:08.604776 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:08.604898 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:08.612135 23142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:08.612258 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:08.612386 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:08.612471 23142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:08.612574 23142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:08.622874 23142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45205,127.0.97.61:36251,127.0.97.60:36403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:08.624193 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:08.625593 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:08.637084 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:08.637145 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:08.637049 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:08.638600 23142 server_base.cc:1034] running on GCE node
I20260926 03:05:08.639319 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:08.640633 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:08.641876 23142 hybrid_clock.cc:648] HybridClock initialized: now 1790391908641807 us; error 40 us; skew 500 ppm
I20260926 03:05:08.642194 23142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:08 dist-test-slave-fccw krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:08.645138 23142 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260926 03:05:08.646591 23142 webserver.cc:460] Webserver started at http://127.0.97.62:45931/ using document root <none> and password file <none>
I20260926 03:05:08.647284 23142 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:08.647413 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:08.647717 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:08.650616 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data/instance:
uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d"
format_stamp: "Formatted at 2026-09-26 03:05:08 on dist-test-slave-fccw"
server_key: "d3ed314c98bcdf2318e500ac445bddc1"
server_key_iv: "cd65d3e432e48debb40a78fb7fc1ea0f"
server_key_version: "encryptionkey@0"
I20260926 03:05:08.651379 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/wal/instance:
uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d"
format_stamp: "Formatted at 2026-09-26 03:05:08 on dist-test-slave-fccw"
server_key: "d3ed314c98bcdf2318e500ac445bddc1"
server_key_iv: "cd65d3e432e48debb40a78fb7fc1ea0f"
server_key_version: "encryptionkey@0"
I20260926 03:05:08.656258 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 03:05:08.659502 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:08.660768 23142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 03:05:08.660939 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/wal
uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d"
format_stamp: "Formatted at 2026-09-26 03:05:08 on dist-test-slave-fccw"
server_key: "d3ed314c98bcdf2318e500ac445bddc1"
server_key_iv: "cd65d3e432e48debb40a78fb7fc1ea0f"
server_key_version: "encryptionkey@0"
I20260926 03:05:08.661196 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:08.678929 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:08.682775 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:08.683041 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:08.701849 23142 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45205
I20260926 03:05:08.701869 23208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45205 every 8 connection(s)
I20260926 03:05:08.703280 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/data/info.pb
I20260926 03:05:08.706580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 23142
I20260926 03:05:08.706774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-0/wal/instance
I20260926 03:05:08.707159 23209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:08.707350 388 external_mini_cluster.cc:1428] Setting key f9c71b66b296f50932cf2a866e71f7eb
I20260926 03:05:08.717097 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:08 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260926 03:05:08.740824 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.721927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.0.97.62:45205)
0926 03:05:08.722587 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:08.722597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:08.722647 (+ 50us) server_negotiation.cc:407] Connection header received
0926 03:05:08.723898 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:08.723908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:08.724205 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:08.724423 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:08.725576 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:08.727686 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:08.729038 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:08.730025 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:08.730227 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:08.733739 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:08.733749 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:08.733764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:08.733798 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:08.736795 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:08.737657 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:08.737663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:08.737665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:08.737797 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:08.738263 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:08.738266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:08.738268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:08.738706 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:08.738871 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:08.739507 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 03:05:08.739771 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260926 03:05:08.740852 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.720596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45205 (local address 127.0.0.1:47364)
0926 03:05:08.721465 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.721517 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:08.723529 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:08.724438 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:08.724443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:08.724978 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:08.725359 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:08.725376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:08.727835 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:08.727839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:08.728882 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:08.728894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:08.730243 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:08.730248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:08.730408 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:08.731059 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:08.731085 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:08.733523 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:08.736933 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:08.736944 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:08.736955 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:08.737536 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:08.737906 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:08.737912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:08.737919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:08.738152 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:08.738850 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:08.738860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:08.739127 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 03:05:08.739394 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 03:05:08.739931 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":43,"thread_start_us":400,"threads_started":1}
I20260926 03:05:08.743817 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:08.744743 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.744406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43018)
0926 03:05:08.744551 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.744647 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 03:05:08.745078 23160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
I20260926 03:05:08.746637 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.746194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43030)
0926 03:05:08.746379 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.746479 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 03:05:08.746853 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36251: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260926 03:05:08.767854 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36251
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45205,127.0.97.61:36251,127.0.97.60:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:05:08.796291 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } attempt: 1
I20260926 03:05:08.797442 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.796980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43036)
0926 03:05:08.797213 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.797303 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 03:05:08.798338 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.798031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43042)
0926 03:05:08.798160 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.798223 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:05:08.798528 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36251: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
I20260926 03:05:08.866818 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } attempt: 2
I20260926 03:05:08.868081 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.867534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43044)
0926 03:05:08.867796 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.867908 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 03:05:08.869140 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.868703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43054)
0926 03:05:08.868845 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.868942 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 03:05:08.869313 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36251: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
I20260926 03:05:08.960608 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } attempt: 3
I20260926 03:05:08.962220 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.961604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43064)
0926 03:05:08.961891 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.962022 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 03:05:08.963366 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:08.962898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43070)
0926 03:05:08.963071 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:08.963180 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:05:08.963557 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36251: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
W20260926 03:05:09.100915 23216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:05:09.101385 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:09.101531 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:09.101708 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 03:05:09.106832 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } attempt: 4
I20260926 03:05:09.108052 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.107553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43074)
0926 03:05:09.107840 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.107927 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 03:05:09.108973 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.108650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43090)
0926 03:05:09.108773 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.108856 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 03:05:09.109201 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36251: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36251: connect: Connection refused (error 111)
W20260926 03:05:09.111387 23216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:09.111482 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:09.111600 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:09.111670 23216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:09.111750 23216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:09.121431 23216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45205,127.0.97.61:36251,127.0.97.60:36403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:09.122731 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:09.124001 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:09.134949 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:09.134985 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:09.134908 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:09.135838 23216 server_base.cc:1034] running on GCE node
I20260926 03:05:09.136530 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:09.137848 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:09.139079 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790391909138991 us; error 36 us; skew 500 ppm
I20260926 03:05:09.139432 23216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:09.142659 23216 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260926 03:05:09.144097 23216 webserver.cc:460] Webserver started at http://127.0.97.61:44269/ using document root <none> and password file <none>
I20260926 03:05:09.144732 23216 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:09.144817 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:09.145164 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:09.147958 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data/instance:
uuid: "848aec5722324db49953e4e87360a696"
format_stamp: "Formatted at 2026-09-26 03:05:09 on dist-test-slave-fccw"
server_key: "a24ff35ab1a79559da4401ab62f85b6f"
server_key_iv: "bfa40063d9c87ac59abe60ea06aa6faa"
server_key_version: "encryptionkey@0"
I20260926 03:05:09.148689 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/wal/instance:
uuid: "848aec5722324db49953e4e87360a696"
format_stamp: "Formatted at 2026-09-26 03:05:09 on dist-test-slave-fccw"
server_key: "a24ff35ab1a79559da4401ab62f85b6f"
server_key_iv: "bfa40063d9c87ac59abe60ea06aa6faa"
server_key_version: "encryptionkey@0"
I20260926 03:05:09.153882 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260926 03:05:09.157312 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:09.158633 23216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:05:09.158828 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/wal
uuid: "848aec5722324db49953e4e87360a696"
format_stamp: "Formatted at 2026-09-26 03:05:09 on dist-test-slave-fccw"
server_key: "a24ff35ab1a79559da4401ab62f85b6f"
server_key_iv: "bfa40063d9c87ac59abe60ea06aa6faa"
server_key_version: "encryptionkey@0"
I20260926 03:05:09.159060 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:09.179648 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:09.184059 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:09.184360 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:09.202284 23216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36251
I20260926 03:05:09.202309 23282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36251 every 8 connection(s)
I20260926 03:05:09.203603 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/data/info.pb
I20260926 03:05:09.207221 23283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:09.212836 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 23216
I20260926 03:05:09.213043 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-1/wal/instance
I20260926 03:05:09.213475 388 external_mini_cluster.cc:1428] Setting key 8865d9709b8dbf73f06e2b8148d27145
I20260926 03:05:09.213758 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:09.235489 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.216304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47368 (local address 127.0.97.62:45205)
0926 03:05:09.216548 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.216556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.217625 (+ 1069us) server_negotiation.cc:407] Connection header received
0926 03:05:09.219497 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.219505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.219604 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.219731 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.221170 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.222299 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.223931 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.224710 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.224845 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.228416 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.228425 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.228432 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.228476 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.232186 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.232874 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.232883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.232890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.233077 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.233457 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.233464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.233470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.233868 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.234053 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.234757 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.235029 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 03:05:09.235998 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.216638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45205 (local address 127.0.0.1:47368)
0926 03:05:09.217452 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.217497 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.219223 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.219910 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.219917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.220430 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.220956 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.220972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.222561 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.222564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.223768 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.223779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.224914 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.224918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.225166 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.225861 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.225882 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.228194 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.232329 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.232335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.232346 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.232739 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.233179 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.233182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.233184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.233338 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.234000 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.234005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.234384 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.234595 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.234949 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":367,"threads_started":1}
I20260926 03:05:09.237753 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:05:09.252419 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.238422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43094)
0926 03:05:09.238642 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.238656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.238729 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.239681 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.239686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.240088 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.240230 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.240238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.242002 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.242005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.242784 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.242794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.243940 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.243944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.244089 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.244614 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.244630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.246603 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.250253 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.250256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.250259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.250599 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.250994 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.250997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.251000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.251128 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.251782 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.251785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.251881 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.251977 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.252171 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2}
I20260926 03:05:09.252976 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.238704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.0.97.61:36251)
0926 03:05:09.239286 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.239292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.239313 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:05:09.239370 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.239378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.239553 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.239697 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.240401 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.241826 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.242950 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.243812 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.243950 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.246810 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.246825 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.246843 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.246892 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.250134 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.250714 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.250725 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.250732 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.250881 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.251220 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.251227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.251233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.251667 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.251843 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.252164 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.252439 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":457,"spinlock_wait_cycles":17664,"thread_start_us":336,"threads_started":1}
I20260926 03:05:09.255041 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:09.256178 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.255764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52842)
0926 03:05:09.255935 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.256045 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 03:05:09.256436 23235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.258015 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.257671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52856)
0926 03:05:09.257851 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.257920 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 03:05:09.258291 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260926 03:05:09.271860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task33M2KT/build/asan/bin/kudu
/tmp/dist-test-task33M2KT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36403
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45205,127.0.97.61:36251,127.0.97.60:36403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:05:09.307629 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 1
I20260926 03:05:09.308864 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.308324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52864)
0926 03:05:09.308557 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.308669 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 03:05:09.309921 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.309541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52870)
0926 03:05:09.309696 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.309783 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:05:09.310132 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.378374 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 2
I20260926 03:05:09.379649 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.379115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52874)
0926 03:05:09.379342 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.379463 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 03:05:09.380692 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.380277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52880)
0926 03:05:09.380439 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.380536 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 03:05:09.380975 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.408469 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } attempt: 5
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:05:09.424091 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.409476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.0.97.61:36251)
0926 03:05:09.410295 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.410303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.410325 (+ 22us) server_negotiation.cc:407] Connection header received
0926 03:05:09.410399 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.410406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.410498 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.410623 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.411640 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.412713 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.414486 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.415138 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.415262 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.418794 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.418798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.418802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.418834 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.421478 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.422271 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.422275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.422277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.422384 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.422774 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.422777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.422779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.423020 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.423132 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.423720 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.423873 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646}
I20260926 03:05:09.424465 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.409471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43102)
0926 03:05:09.409820 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.409844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.410009 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.410686 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.410693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.411261 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.411473 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.411484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.412905 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.412911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.414286 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.414302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.415278 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.415283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.415428 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.416070 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.416088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.418612 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.421625 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.421644 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.421648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.422167 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.422484 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.422489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.422493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.422666 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.423138 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.423146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.423456 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.423653 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.424007 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260926 03:05:09.425592 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:09.426699 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.426268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52892)
0926 03:05:09.426445 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.426536 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 03:05:09.427709 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.427334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52908)
0926 03:05:09.427476 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.427571 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 03:05:09.427992 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.472260 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 3
I20260926 03:05:09.473497 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.472930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52914)
0926 03:05:09.473194 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.473305 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 03:05:09.474707 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.474144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52926)
0926 03:05:09.474411 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.474530 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260926 03:05:09.474937 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.479190 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 1
I20260926 03:05:09.480127 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.479680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52930)
0926 03:05:09.479871 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.479960 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 03:05:09.481178 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.480781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52934)
0926 03:05:09.480943 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.481042 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:05:09.481410 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.549628 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 2
I20260926 03:05:09.550890 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.550296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52936)
0926 03:05:09.550550 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.550663 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 03:05:09.551962 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.551575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52948)
0926 03:05:09.551727 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.551821 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 03:05:09.552239 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
W20260926 03:05:09.602062 23290 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 03:05:09.602407 23290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:05:09.602527 23290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:05:09.602653 23290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:05:09.610874 23290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:05:09.610972 23290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:05:09.611068 23290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:05:09.611157 23290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:05:09.611253 23290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:05:09.618156 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 4
I20260926 03:05:09.619436 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.618891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52956)
0926 03:05:09.619142 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.619247 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 03:05:09.620606 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.620149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52966)
0926 03:05:09.620326 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.620431 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 03:05:09.620929 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.622265 23290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45205,127.0.97.61:36251,127.0.97.60:36403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:36403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 02:01:27 UTC on ccycloud.quasar-eeefwb.root.comops.site
build id 4673635
ASAN enabled
I20260926 03:05:09.623657 23290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:05:09.625072 23290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:05:09.639256 23296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:09.639398 23295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:05:09.639869 23298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:05:09.642467 23290 server_base.cc:1034] running on GCE node
I20260926 03:05:09.643319 23290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:05:09.644824 23290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:05:09.646056 23290 hybrid_clock.cc:648] HybridClock initialized: now 1790391909645964 us; error 69 us; skew 500 ppm
I20260926 03:05:09.646396 23290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:05:09.649467 23290 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260926 03:05:09.651053 23290 webserver.cc:460] Webserver started at http://127.0.97.60:45871/ using document root <none> and password file <none>
I20260926 03:05:09.651746 23290 fs_manager.cc:359] Metadata directory not provided
I20260926 03:05:09.651855 23290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:05:09.652170 23290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:05:09.655071 23290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data/instance:
uuid: "88b4fcdacb034262894dc766d298ce15"
format_stamp: "Formatted at 2026-09-26 03:05:09 on dist-test-slave-fccw"
server_key: "3ede3c68f41866e472c7c68eecdd1832"
server_key_iv: "aa4a73393c1ed8c67fef0be7128085f8"
server_key_version: "encryptionkey@0"
I20260926 03:05:09.655827 23290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/wal/instance:
uuid: "88b4fcdacb034262894dc766d298ce15"
format_stamp: "Formatted at 2026-09-26 03:05:09 on dist-test-slave-fccw"
server_key: "3ede3c68f41866e472c7c68eecdd1832"
server_key_iv: "aa4a73393c1ed8c67fef0be7128085f8"
server_key_version: "encryptionkey@0"
I20260926 03:05:09.660861 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 3
I20260926 03:05:09.661033 23290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 03:05:09.662132 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.661546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52982)
0926 03:05:09.661799 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.661911 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 03:05:09.663537 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.662933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52984)
0926 03:05:09.663248 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.663364 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260926 03:05:09.663842 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:36403: Network error: Client connection negotiation failed: client connection to 127.0.97.60:36403: connect: Connection refused (error 111)
I20260926 03:05:09.665306 23305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:05:09.666729 23290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 03:05:09.666905 23290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data,/tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/wal
uuid: "88b4fcdacb034262894dc766d298ce15"
format_stamp: "Formatted at 2026-09-26 03:05:09 on dist-test-slave-fccw"
server_key: "3ede3c68f41866e472c7c68eecdd1832"
server_key_iv: "aa4a73393c1ed8c67fef0be7128085f8"
server_key_version: "encryptionkey@0"
I20260926 03:05:09.667105 23290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:05:09.701725 23290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:05:09.706133 23290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:05:09.706415 23290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:05:09.726058 23290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:36403
I20260926 03:05:09.726092 23356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:36403 every 8 connection(s)
I20260926 03:05:09.727440 23290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/data/info.pb
I20260926 03:05:09.731172 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:05:09.736016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task33M2KT/build/asan/bin/kudu as pid 23290
I20260926 03:05:09.736207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task33M2KT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790391557890803-388-0/minicluster-data/master-2/wal/instance
I20260926 03:05:09.736892 388 external_mini_cluster.cc:1428] Setting key 14f41642de324cce58edeca4c6f73218
I20260926 03:05:09.737874 23357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:05:09.775825 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.741546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47384 (local address 127.0.97.62:45205)
0926 03:05:09.754862 (+ 13316us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.754870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.754891 (+ 21us) server_negotiation.cc:407] Connection header received
0926 03:05:09.756087 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.756092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.756179 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.756300 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.758095 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.759578 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.761539 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.761971 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.762098 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.766738 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.766743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.766746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.766776 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.770684 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.772029 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.772034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.772040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.772206 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.772743 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.772752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.772757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.773213 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.773390 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.775186 (+ 1796us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.775423 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13170,"thread_start_us":13060,"threads_started":1}
I20260926 03:05:09.776101 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.739198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52986 (local address 127.0.97.60:36403)
0926 03:05:09.753850 (+ 14652us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.753863 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.753935 (+ 72us) server_negotiation.cc:407] Connection header received
0926 03:05:09.755840 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.755882 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.756288 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.756553 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.758279 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.759442 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.761210 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.762326 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.762577 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.767871 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.767883 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.767900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.767942 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.771899 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.772669 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.772677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.772681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.772806 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.773212 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.773216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.773220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.773823 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.774192 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.774507 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.774805 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":14367,"thread_start_us":14903,"threads_started":1}
I20260926 03:05:09.776877 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.741593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45205 (local address 127.0.0.1:47384)
0926 03:05:09.753585 (+ 11992us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.753649 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.755814 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.756386 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.756399 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.757319 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.757824 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.757870 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.759752 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.759758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.761406 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.761417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.762217 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.762226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.762477 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.763517 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.763557 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.766494 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.770858 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.770873 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.770890 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.771873 (+ 983us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.772326 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.772333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.772339 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.772577 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.773355 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.773363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.773865 (+ 502us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.774179 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.774691 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11656,"mutex_wait_us":93,"spinlock_wait_cycles":18176,"thread_start_us":12069,"threads_started":1}
I20260926 03:05:09.778491 23357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260926 03:05:09.793196 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.779290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.0.97.61:36251)
0926 03:05:09.779598 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.779607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.779864 (+ 257us) server_negotiation.cc:407] Connection header received
0926 03:05:09.780023 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.780027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.780090 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.780197 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.780962 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.782241 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.783601 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.784022 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.784149 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.787566 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.787570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.787573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.787603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.790537 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.791297 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.791305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.791308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.791445 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.791801 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.791806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.791810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.792092 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.792256 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.792693 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.792861 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260926 03:05:09.793443 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.779194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36251 (local address 127.0.0.1:43112)
0926 03:05:09.779752 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.779784 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.779908 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.780217 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.780221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.780589 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.780787 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.780801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.782410 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.782415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.783465 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.783476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.784167 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.784172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.784349 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.784919 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.784945 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.787380 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.790704 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.790710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.790714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.791186 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.791551 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.791556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.791559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.791690 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.792238 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.792246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.792442 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.792619 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.792947 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430}
I20260926 03:05:09.794390 23357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:05:09.799419 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.779516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.0.97.62:45205)
0926 03:05:09.779688 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.779691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.780396 (+ 705us) server_negotiation.cc:407] Connection header received
0926 03:05:09.780455 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.780462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.780546 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.780647 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.781887 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.782797 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.784982 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.785655 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.785781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.791282 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.791291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.791298 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.791345 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.796536 (+ 5191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.797267 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.797273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.797276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.797432 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.797840 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.797845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.797849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.798205 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.798493 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.798775 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.798991 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:05:09.811247 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.795230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:52994)
0926 03:05:09.795580 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.795602 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.795714 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.796047 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.796054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.796636 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.796851 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.796865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.798197 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.798205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.800465 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.800482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.801602 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.801608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.801756 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.802312 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.802336 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.804648 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.807992 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.807999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.808004 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.808484 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.809263 (+ 779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.809270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.809276 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.809460 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.810163 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.810170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.810356 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.810529 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.810826 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":50}
I20260926 03:05:09.811650 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.795142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.0.97.60:36403)
0926 03:05:09.795371 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.795377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.795683 (+ 306us) server_negotiation.cc:407] Connection header received
0926 03:05:09.795831 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.795836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.795902 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.796004 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.797044 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.798062 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.800659 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.801307 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.801434 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.804810 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.804820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.804823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.804853 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.807876 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.808914 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.808922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.808927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.809120 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.809585 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.809592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.809598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.810014 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.810165 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.810664 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.811019 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":58}
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260926 03:05:09.818372 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.801375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.0.97.61:36251)
0926 03:05:09.801564 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.801572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.801591 (+ 19us) server_negotiation.cc:407] Connection header received
0926 03:05:09.801645 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.801652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.801734 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.801852 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.803678 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.805066 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.807174 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.807994 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.808127 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.813335 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.813341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.813346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.813382 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.815828 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.816487 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.816493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.816497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.816618 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.816973 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.816980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.816983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.817274 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.817537 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.817784 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.818011 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 03:05:09.818638 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: Bootstrap starting.
I20260926 03:05:09.822579 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:05:09.823411 23357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:09.824946 23357 log.cc:824] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:09.824885 23184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47390
I20260926 03:05:09.828425 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: No bootstrap required, opened a new log
I20260926 03:05:09.841061 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 4
I20260926 03:05:09.841929 23357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:09.842342 23357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b4fcdacb034262894dc766d298ce15, State: Initialized, Role: FOLLOWER
I20260926 03:05:09.842850 23357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:09.844957 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:09.845566 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:09.845942 23357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260926 03:05:09.857050 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.841703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:53004)
0926 03:05:09.841941 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.841960 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.842099 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.842515 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.842521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.843020 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.843238 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.843249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.844330 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.844344 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.845854 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.845871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.846691 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.846697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.846920 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.847628 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.847652 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.850498 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.854144 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.854149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.854153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.854664 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.855075 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.855080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.855083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.855270 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.856042 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.856049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.856214 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.856343 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.856640 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 03:05:09.858302 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.841752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.0.97.60:36403)
0926 03:05:09.841941 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.841948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.842024 (+ 76us) server_negotiation.cc:407] Connection header received
0926 03:05:09.842266 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.842291 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.842372 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.842479 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.843430 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.844186 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.845988 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.846529 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.846680 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.850680 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.850685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.850689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.850716 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.854009 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.854793 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.854800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.854803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.854948 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.855396 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.855404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.855407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.855881 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.857474 (+ 1593us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.857645 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.857918 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 03:05:09.864027 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d: Bootstrap starting.
I20260926 03:05:09.867913 23209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:09.868889 23209 log.cc:824] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:09.871770 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d: No bootstrap required, opened a new log
W20260926 03:05:09.872051 23375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:05:09.872169 23375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:05:09.867733 23290 master_runner.cc:186] Error getting master registration for 127.0.97.62:45205: OK, instance_id { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" instance_seqno: 1790391908696640 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:05:09.879225 23290 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:05:09.879035 23209 raft_consensus.cc:348] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:09.879451 23209 raft_consensus.cc:729] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23f2ebd7ddc48dcb5325dc5b2ee207d, State: Initialized, Role: FOLLOWER
I20260926 03:05:09.879930 23209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:09.880856 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:09.881260 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:09.882184 23209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:09.882261 23184 raft_consensus.cc:482] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 03:05:09.882471 23184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:09.885332 23184 raft_consensus.cc:504] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:09.887552 23332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "88b4fcdacb034262894dc766d298ce15"
I20260926 03:05:09.888053 23332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:09.891776 23332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23f2ebd7ddc48dcb5325dc5b2ee207d in term 1.
I20260926 03:05:09.892153 23184 leader_election.cc:288] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [CANDIDATE]: Term 1 election: Requested vote from peers 848aec5722324db49953e4e87360a696 (127.0.97.61:36251), 88b4fcdacb034262894dc766d298ce15 (127.0.97.60:36403)
I20260926 03:05:09.892453 23161 leader_election.cc:302] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b4fcdacb034262894dc766d298ce15, a23f2ebd7ddc48dcb5325dc5b2ee207d; no voters:
I20260926 03:05:09.893026 23378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:05:09.893704 23378 raft_consensus.cc:686] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [term 1 LEADER]: Becoming Leader. State: Replica: a23f2ebd7ddc48dcb5325dc5b2ee207d, State: Running, Role: LEADER
I20260926 03:05:09.894213 23378 consensus_queue.cc:237] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:09.897432 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a23f2ebd7ddc48dcb5325dc5b2ee207d. Latest consensus state: current_term: 1 leader_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:09.897800 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:09.900280 23394 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 03:05:09.905771 23393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:05:09.920158 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } attempt: 5
I20260926 03:05:09.905259 23142 master_runner.cc:186] Error getting master registration for 127.0.97.61:36251: OK, instance_id { permanent_uuid: "848aec5722324db49953e4e87360a696" instance_seqno: 1790391909197280 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 03:05:09.927137 23142 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 03:05:09.928565 23393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 03:05:09 dist-test-slave-fccw krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790391909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260926 03:05:09.935760 23163 tablet.cc:2369] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:05:09.954329 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.920812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:36403 (local address 127.0.0.1:53014)
0926 03:05:09.921266 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 03:05:09.921288 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 03:05:09.921419 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:05:09.922172 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:05:09.922177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:05:09.922786 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:05:09.922998 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.923011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.924540 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.924546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.926133 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:05:09.926149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.927436 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.927442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:05:09.927642 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.928557 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:05:09.928613 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:05:09.940210 (+ 11597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:05:09.944923 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.944931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.944936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.948335 (+ 3399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.949733 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:05:09.949740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:05:09.949745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:05:09.950791 (+ 1046us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:05:09.952707 (+ 1916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:05:09.952713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:05:09.953471 (+ 758us) client_negotiation.cc:769] Sending connection context
0926 03:05:09.953640 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 03:05:09.953949 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260926 03:05:09.955269 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.921241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.97.60:36403)
0926 03:05:09.921529 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.921535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.921558 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:05:09.921620 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.921625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.921703 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.921953 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.923185 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.924362 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.926314 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.927050 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.927173 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.940803 (+ 13630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.940811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.940815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.940859 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.944714 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.948869 (+ 4155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.948877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.948881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.949192 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.951406 (+ 2214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.951413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.951417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.952087 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.952361 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.953919 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.954342 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
W20260926 03:05:09.958844 23311 tablet.cc:2369] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:05:09.961975 23332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [term 1 FOLLOWER]: Refusing update from remote peer a23f2ebd7ddc48dcb5325dc5b2ee207d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:05:09.962982 23378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:05:09.968909 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696: Bootstrap starting.
I20260926 03:05:09.974397 23283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696: Neither blocks nor log segments found. Creating new log.
I20260926 03:05:09.975773 23283 log.cc:824] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696: Log is configured to *not* fsync() on all Append() calls
I20260926 03:05:09.988353 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.929444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.0.97.60:36403)
0926 03:05:09.955914 (+ 26470us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.955919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.955963 (+ 44us) server_negotiation.cc:407] Connection header received
0926 03:05:09.956041 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.956046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.956183 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.956382 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.960132 (+ 3750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.973365 (+ 13233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.975471 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.976164 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.979847 (+ 3683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.980048 (+ 201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.980054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.980059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.980089 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.984205 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.985205 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.985209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.985213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.985395 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:09.985887 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:09.985892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:09.985895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:09.986237 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:09.987470 (+ 1233us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:09.987663 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 03:05:09.987925 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26309,"thread_start_us":33428,"threads_started":1}
I20260926 03:05:10.003486 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a23f2ebd7ddc48dcb5325dc5b2ee207d. Latest consensus state: current_term: 1 leader_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:10.003727 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.939037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.0.97.62:45205)
0926 03:05:09.939204 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.939213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.962297 (+ 23084us) server_negotiation.cc:407] Connection header received
0926 03:05:09.976850 (+ 14553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.976858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.976954 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.977061 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.979391 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.980532 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.991011 (+ 10479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.991747 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.991858 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:09.995683 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:09.995690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:09.995695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:09.995750 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:09.999634 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:10.000450 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:10.000455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:10.000458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:10.000644 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:10.001125 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:10.001130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:10.001133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:10.001528 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:10.001818 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.002964 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.003241 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 03:05:10.003818 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:10.004691 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696: No bootstrap required, opened a new log
I20260926 03:05:10.006016 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:10.006302 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:10.011071 23393 catalog_manager.cc:1347] Generated new cluster ID: f9e35e006d7f4d46ac14ab4eccf8ecff
I20260926 03:05:10.011204 23393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:05:10.017449 23402 mvcc.cc:205] Tried to move back new op lower bound from 7333445263178637312 to 7333445262932426752. Current Snapshot: MvccSnapshot[applied={T|T < 7333445263178637312}]
I20260926 03:05:10.024659 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:10.024916 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:10.036298 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:09.962838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.0.97.61:36251)
0926 03:05:09.963004 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:09.963013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:09.985033 (+ 22020us) server_negotiation.cc:407] Connection header received
0926 03:05:09.985803 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:09.985809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:09.985896 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:09.985969 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:09.995660 (+ 9691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.996835 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.998709 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:09.999455 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:09.999592 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.027119 (+ 27527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:05:10.027126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:05:10.027131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:05:10.027178 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:05:10.030778 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:10.033992 (+ 3214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:10.034000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:10.034006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:10.034158 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:05:10.034593 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:05:10.034598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:05:10.034603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:05:10.034934 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:05:10.035147 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.035703 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.035931 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:05:10.042702 23283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:10.043071 23283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848aec5722324db49953e4e87360a696, State: Initialized, Role: FOLLOWER
I20260926 03:05:10.047008 23283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } }
I20260926 03:05:10.060259 23393 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:05:10.063031 23283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:05:10.063246 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "848aec5722324db49953e4e87360a696"
I20260926 03:05:10.063503 23257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:05:10.064554 23393 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:05:10.066948 23257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [term 1 FOLLOWER]: Refusing update from remote peer a23f2ebd7ddc48dcb5325dc5b2ee207d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:05:10.070080 23258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a23f2ebd7ddc48dcb5325dc5b2ee207d in term 1.
I20260926 03:05:10.075136 23384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [LEADER]: Connected to new peer: Peer: permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:05:10.058010 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:10.076224 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:10.081009 23393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d: Generated new TSK 0
I20260926 03:05:10.082224 23393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:05:10.118268 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:10.118588 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23f2ebd7ddc48dcb5325dc5b2ee207d [sys.catalog]: This master's current role is: LEADER
I20260926 03:05:10.119874 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:10.125358 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:10.129786 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a23f2ebd7ddc48dcb5325dc5b2ee207d. Latest consensus state: current_term: 1 leader_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a23f2ebd7ddc48dcb5325dc5b2ee207d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45205 } } peers { permanent_uuid: "848aec5722324db49953e4e87360a696" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36251 } } peers { permanent_uuid: "88b4fcdacb034262894dc766d298ce15" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 36403 } } }
I20260926 03:05:10.130168 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:05:10.190899 23427 catalog_manager.cc:1259] Loaded cluster ID: f9e35e006d7f4d46ac14ab4eccf8ecff
I20260926 03:05:10.191015 23427 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696: loading cluster ID for follower catalog manager: success
W20260926 03:05:10.218439 23361 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.196318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.0.97.62:45205)
0926 03:05:10.196504 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.196510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.199847 (+ 3337us) server_negotiation.cc:407] Connection header received
0926 03:05:10.200851 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.200857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.200940 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.201049 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.218175 (+ 17126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47408: BlockingRecv error: recv got EOF from 127.0.0.1:47408 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 03:05:10.223057 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.197247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.0.97.60:36403)
0926 03:05:10.197416 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.197423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.219231 (+ 21808us) server_negotiation.cc:407] Connection header received
0926 03:05:10.221551 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.221557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.221653 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.221728 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.222826 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53022: BlockingRecv error: recv got EOF from 127.0.0.1:53022 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:05:10.225363 23427 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696: acquiring CA information for follower catalog manager: success
I20260926 03:05:10.228132 23427 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 848aec5722324db49953e4e87360a696: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:05:10.229789 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.219767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.0.97.61:36251)
0926 03:05:10.219961 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.219972 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.220919 (+ 947us) server_negotiation.cc:407] Connection header received
0926 03:05:10.228407 (+ 7488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.228413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.228488 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.228557 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.229584 (+ 1027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43142: BlockingRecv error: recv got EOF from 127.0.0.1:43142 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 03:05:10.279330 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.263765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.97.61:36251)
0926 03:05:10.263981 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.263987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.271449 (+ 7462us) server_negotiation.cc:407] Connection header received
0926 03:05:10.271794 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.271801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.272033 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.272173 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.272887 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.274093 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.276406 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.277104 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.277201 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.277899 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.278529 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.278642 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.278887 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.278976 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260926 03:05:10.325717 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.322667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.0.97.60:36403)
0926 03:05:10.322885 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.322890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.323792 (+ 902us) server_negotiation.cc:407] Connection header received
0926 03:05:10.324128 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.324135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.324268 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.324348 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.325462 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53036: BlockingRecv error: recv got EOF from 127.0.0.1:53036 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 03:05:10.378041 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.365623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.0.97.61:36251)
0926 03:05:10.365826 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.365832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.371025 (+ 5193us) server_negotiation.cc:407] Connection header received
0926 03:05:10.371298 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.371305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.371411 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.371528 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.372234 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.373470 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.375555 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.376257 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.376363 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.377114 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.377281 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.377350 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.377603 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.377694 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 03:05:10.421969 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.412709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.0.97.60:36403)
0926 03:05:10.412931 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.412938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.419833 (+ 6895us) server_negotiation.cc:407] Connection header received
0926 03:05:10.420358 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.420365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.420472 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.420563 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.421687 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53050: BlockingRecv error: recv got EOF from 127.0.0.1:53050 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 03:05:10.480505 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.461145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.0.97.61:36251)
0926 03:05:10.461350 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.461356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.473701 (+ 12345us) server_negotiation.cc:407] Connection header received
0926 03:05:10.473957 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.473963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.474067 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.474161 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.474896 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.476052 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.478114 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.478906 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.479010 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.479632 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.479770 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.479846 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.480062 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.480145 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 03:05:10.528638 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.519608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.0.97.60:36403)
0926 03:05:10.519807 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.519814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.526624 (+ 6810us) server_negotiation.cc:407] Connection header received
0926 03:05:10.527113 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.527121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.527214 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.527314 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.528372 (+ 1058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53060: BlockingRecv error: recv got EOF from 127.0.0.1:53060 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:05:10.557736 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.550851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43168 (local address 127.0.97.61:36251)
0926 03:05:10.551072 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.551081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.551591 (+ 510us) server_negotiation.cc:407] Connection header received
0926 03:05:10.551787 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.551794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.551940 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.552050 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.552525 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.553730 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.555631 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.556362 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.556488 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.556932 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.557103 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.557193 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.557356 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.557432 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260926 03:05:10.578790 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.576823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53068 (local address 127.0.97.60:36403)
0926 03:05:10.577134 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.577141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.577426 (+ 285us) server_negotiation.cc:407] Connection header received
0926 03:05:10.577626 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.577632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.577724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.577848 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.578556 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53068: BlockingRecv error: recv got EOF from 127.0.0.1:53068 (error 108)
Metrics: {"server-negotiator.queue_time_us":152}
I20260926 03:05:10.595713 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.589849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43182 (local address 127.0.97.61:36251)
0926 03:05:10.590046 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.590052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.590396 (+ 344us) server_negotiation.cc:407] Connection header received
0926 03:05:10.590549 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.590554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.590654 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.590776 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.591128 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.592284 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.593726 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.594535 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.594679 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.595005 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.595137 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.595234 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.595321 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.595397 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 03:05:10.603914 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.601856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.0.97.60:36403)
0926 03:05:10.602057 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.602063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.602483 (+ 420us) server_negotiation.cc:407] Connection header received
0926 03:05:10.602652 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.602656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.602723 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.602804 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.603696 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53082: BlockingRecv error: recv got EOF from 127.0.0.1:53082 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 03:05:10.629310 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.623646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.0.97.61:36251)
0926 03:05:10.623858 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.623865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.624140 (+ 275us) server_negotiation.cc:407] Connection header received
0926 03:05:10.624301 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.624308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.624396 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.624501 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.624877 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.626137 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.627424 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.628064 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.628188 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.628563 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.628714 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.628831 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.628918 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.629023 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 03:05:10.637452 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.635315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.0.97.60:36403)
0926 03:05:10.635638 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.635644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.636093 (+ 449us) server_negotiation.cc:407] Connection header received
0926 03:05:10.636284 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.636288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.636358 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.636461 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.637174 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53098: BlockingRecv error: recv got EOF from 127.0.0.1:53098 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 03:05:10.681806 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.675927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.97.61:36251)
0926 03:05:10.676182 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.676188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.676489 (+ 301us) server_negotiation.cc:407] Connection header received
0926 03:05:10.676706 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.676710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.676780 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.676861 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.677335 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.678075 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.679978 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.680387 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.680493 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.680924 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.681084 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.681152 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.681337 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.681409 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 03:05:10.689230 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.687281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.0.97.60:36403)
0926 03:05:10.687494 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.687500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.687844 (+ 344us) server_negotiation.cc:407] Connection header received
0926 03:05:10.687988 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.687991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.688059 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.688136 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.688961 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53114: BlockingRecv error: recv got EOF from 127.0.0.1:53114 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 03:05:10.762135 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.756621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.0.97.61:36251)
0926 03:05:10.756924 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.756931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.757348 (+ 417us) server_negotiation.cc:407] Connection header received
0926 03:05:10.757574 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.757579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.757657 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.757780 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.758237 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.759055 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.760458 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.760872 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.761029 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.761418 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.761514 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.761610 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.761767 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.761843 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260926 03:05:10.769490 23360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:05:10.767614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.0.97.60:36403)
0926 03:05:10.767765 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.767770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.768173 (+ 403us) server_negotiation.cc:407] Connection header received
0926 03:05:10.768324 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.768328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.768397 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.768480 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:05:10.769226 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53128: BlockingRecv error: recv got EOF from 127.0.0.1:53128 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 03:05:10.874045 23375 catalog_manager.cc:1259] Loaded cluster ID: f9e35e006d7f4d46ac14ab4eccf8ecff
I20260926 03:05:10.874140 23375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: loading cluster ID for follower catalog manager: success
I20260926 03:05:10.876734 23375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: acquiring CA information for follower catalog manager: success
I20260926 03:05:10.878491 23375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88b4fcdacb034262894dc766d298ce15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:05:10.905543 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.900895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.0.97.61:36251)
0926 03:05:10.901114 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.901119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.901415 (+ 296us) server_negotiation.cc:407] Connection header received
0926 03:05:10.901562 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.901566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.901632 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.901721 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.902071 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.902733 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.903971 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.904390 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.904498 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.904921 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.905097 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.905167 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.905286 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.905345 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 03:05:10.916422 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.910827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.0.97.60:36403)
0926 03:05:10.911055 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.911061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.911404 (+ 343us) server_negotiation.cc:407] Connection header received
0926 03:05:10.911637 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.911648 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.911799 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.911922 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.912387 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.913118 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.914368 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.914773 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.914904 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.915368 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.915870 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.916028 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.916094 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.916167 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 03:05:10.928365 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.921866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.0.97.60:36403)
0926 03:05:10.922021 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.922026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.922956 (+ 930us) server_negotiation.cc:407] Connection header received
0926 03:05:10.923213 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.923220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.923310 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.923379 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.924122 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.924803 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.926535 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.926981 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.927106 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.927642 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.927735 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.927834 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.927998 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.928074 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:05:10.930469 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.921775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.97.62:45205)
0926 03:05:10.922399 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.922408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.922751 (+ 343us) server_negotiation.cc:407] Connection header received
0926 03:05:10.923095 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.923104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.923342 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.923496 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.924418 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.925180 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.927367 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.928033 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.928147 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.929506 (+ 1359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.929892 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.930015 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.930148 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.930229 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":343,"threads_started":1}
I20260926 03:05:10.930655 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:05:10.922922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.0.97.61:36251)
0926 03:05:10.923103 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 03:05:10.923111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:05:10.923670 (+ 559us) server_negotiation.cc:407] Connection header received
0926 03:05:10.923876 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:05:10.923884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:05:10.923983 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:05:10.924081 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:05:10.925684 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.926862 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.928789 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:05:10.929477 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:05:10.929594 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:05:10.929995 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:05:10.930101 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:05:10.930198 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:05:10.930312 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 03:05:10.930385 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 03:05:10.935328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 23142
I20260926 03:05:10.936278 23204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:11.045589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 23142
W20260926 03:05:11.066699 23309 connection.cc:535] server connection from 127.0.0.1:53004 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:05:11.067143 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 23216
I20260926 03:05:11.068195 23278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:11.180500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 23216
I20260926 03:05:11.200276 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task33M2KT/build/asan/bin/kudu pid 23290
I20260926 03:05:11.201364 23352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 03:05:11.307986 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task33M2KT/build/asan/bin/kudu with pid 23290
2026-09-26T03:05:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4384 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9200 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (353380 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST